Celery: supervisord

conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
[program:epm-tasks]
directory=/home/ubuntu/projects/epm/web
user=ubuntu
command=/home/ubuntu/.anyenv/envs/pyenv/versions/coresys/bin/celery -A app worker -l INFO -f /home/ubuntu/projects/epm/logs/tasks.log
autostart=true
autorestart=true
;stdout_logfile=syslog
;stderr_logfile=syslog
numprocs=1
startsecs=10
stopwaitsecs = 600
killasgroup=true

-f オプションで指定すると user の所有権でファイル作成
stdout_logfile / stderr_logfile で指定するとsupervisord ユーザー(root)の所有権で作成

log rotation

It seems that you are using supervisord to manage Celery processes and you want to rotate the log files for Celery on a daily basis. One way to do this is to use logrotate, a tool that can rotate and compress log files according to a configuration file². To use logrotate, you need to do the following steps:

Create a configuration file for logrotate that specifies the log files you want to rotate, the frequency of rotation, the number of rotated files to keep, and any other options you want. For example, you can create a file called /etc/logrotate.d/celery with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
/var/log/supervisor/celery.log
/var/log/supervisor/celery_err.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    postrotate
        supervisorctl signal USR1 celery >/dev/null 2>&1 || true
    endscript
}

This configuration tells logrotate to rotate the celery.log and celery_err.log files daily, keep 7 rotated files, compress them, ignore any missing or empty files, and send a USR1 signal to the celery process after rotation¹.

Make sure that logrotate is running as a cron job at the desired frequency. By default, logrotate runs daily as part of the cron.daily script². If you want to run it hourly, you need to create a symlink from /etc/cron.hourly to /etc/cron.daily/logrotate².
Make sure that supervisord is configured to use a single log file for each Celery process and not rotate them by itself. You can do this by setting the following options in your supervisord.conf file:

1
2
3
4
5
6
7
[program:celery]
...
stdout_logfile = /var/log/supervisor/celery.log
stdout_logfile_maxbytes = 0
stderr_logfile = /var/log/supervisor/celery_err.log
stderr_logfile_maxbytes = 0
...

This configuration tells supervisord to write the standard output and standard error of Celery to separate log files and disable its own rotation mechanism³.

I hope this helps. Do you have any other questions?

ソース: Bing との会話 2023/4/12

(1) ログローテートソフトウエア logrotate についてまとめ - Qiita. https://qiita.com/shotets/items/e13e1d1739eaea7b1ff9 アクセス日時 2023/4/12.
(2) Supervisorで日次のログローテートを行う方法 - Qiita. https://qiita.com/tmiki/items/525bb26699ecca4a182a アクセス日時 2023/4/12.
(3) django - Supervisord / Celery output logs - Stack Overflow. https://stackoverflow.com/questions/31556238/supervisord-celery-output-logs アクセス日時 2023/4/12.