Background Jobs in Flask: Celery and Flask-Scheduler

Introduction

Background jobs allow you to offload time-consuming tasks from the main application, improving performance and user experience. Flask can integrate with tools like Celery for task queues and Flask-Scheduler for periodic jobs.

This article explains step-by-step how to implement background jobs in Flask using Celery and Flask-Scheduler.

Part 1: Task Queues with Celery

Step 1: Install Celery and a Message Broker

Install Celery and a message broker like Redis:

            pip install celery redis
        

Ensure Redis is installed and running:

            sudo apt update
            sudo apt install redis
            sudo systemctl start redis
        

Step 2: Configure Celery in Your Flask App

Create a celery_app.py file to configure Celery:

            from celery import Celery

            def make_celery(app):
                celery = Celery(
                    app.import_name,
                    backend=app.config['CELERY_RESULT_BACKEND'],
                    broker=app.config['CELERY_BROKER_URL']
                )
                celery.conf.update(app.config)
                return celery
        

Update your Flask application (app.py):

            from flask import Flask
            from celery_app import make_celery

            app = Flask(__name__)
            app.config['CELERY_BROKER_URL'] = 'redis://localhost:6379/0'
            app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:6379/0'

            celery = make_celery(app)

            @app.route('/long-task')
            def long_task():
                task = background_task.apply_async()
                return f"Task ID: {task.id}"

            @celery.task
            def background_task():
                import time
                time.sleep(10)  # Simulate a long task
                return "Task completed!"
        

Step 3: Start Celery and Test

Start the Celery worker:

            celery -A app.celery worker --loglevel=info
        

Run the Flask application and access /long-task. Celery processes the task in the background.

Part 2: Periodic Jobs with Flask-Scheduler

Step 1: Install Flask-Scheduler

Install Flask-Scheduler for scheduling cron-like tasks:

            pip install flask-apscheduler
        

Step 2: Configure Flask-Scheduler

Add Flask-Scheduler to your Flask app:

            from flask import Flask
            from flask_apscheduler import APScheduler

            app = Flask(__name__)
            scheduler = APScheduler()

            def scheduled_task():
                print("Running scheduled task")

            if __name__ == "__main__":
                app.config['SCHEDULER_API_ENABLED'] = True
                scheduler.init_app(app)
                scheduler.start()

                scheduler.add_job(
                    id='Scheduled Task',
                    func=scheduled_task,
                    trigger='interval',
                    seconds=10
                )
                app.run(debug=True)
        

Step 3: Test the Scheduled Task

Run the Flask application. The task logs "Running scheduled task" every 10 seconds to the console.

Conclusion

Using Celery and Flask-Scheduler, you can implement both ad-hoc and periodic background jobs in Flask. This improves the scalability and functionality of your application by efficiently handling tasks.