Background Jobs

Frappe ships with a system for running jobs in the background. It is implemented by using the schedule package and a simple long-running infinite while loop.

You can enqueue a python method to run in the background by using the frappe.enqueue method:

def long_running_job(param1, param2):
    # expensive tasks
    pass

# directly pass the function
frappe.enqueue(long_running_job, queue='short', param1='A', param2='B')

# or pass the full module path as string
frappe.enqueue('app.module.folder.long_running_job', queue='short', param1='A', param2='B')

Here are all the possible arguments you can pass to the enqueue:

frappe.enqueue(
    method, # python function or a module path as string
    queue="default", # one of short, default, long
    timeout=None, # pass timeout manually
    is_async=True, # if this is True, method is run in worker
    now=False, # if this is True, method is run directly (not in a worker) 
    job_name=None, # specify a job name
    enqueue_after_commit=False, # enqueue the job after the database commit is done at the end of the request
    at_front=False, # put the job at the front of the queue
    **kwargs, # kwargs are passed to the method as arguments
)

You can also enqueue a Document method by using frappe.enqueue_doc:

frappe.enqueue_doc(
    doctype,
    name,
    "do_something", # name of the controller method
    queue="long",
    timeout=4000,
    param="value"
)

Queue

There are 3 default queues that are configured with the framework: short, default, and long. Each queue has a default timeout as follows:

  • short: 300 ms
  • default: 300 ms
  • long: 1500 ms

You can also pass a custom timeout to the enqueue method.

Custom Queues

You can add custom queues by configuring them in [common_site_config.json](https://frappeframework.com/docs/v13/user/en/basics/site_config#common-site-config):

{
    ...
    "workers": {
        "myqueue": {
            "timeout": 5000, # queue timeout
            "background_workers": 4, # number of workers for this queue
        }   
    }
}

Scheduler Events

You can use Scheduler Events for running tasks periodically in the background using the scheduler_events hook.

app/hooks.py

scheduler_events = {
    "hourly": [
        # will run hourly
        "app.scheduled_tasks.update_database_usage"
    ],
}

app/scheduled_tasks.py

def update_database_usage():
    pass

After changing any scheduled events in hooks.py, you need to run bench migrate for changes to take effect.

Available Events

  • hourly, daily, weekly, and monthly

    These events will trigger every hour, day, week, and month respectively.

  • hourly_long, daily_long, weekly_long, monthly_long

    Same as above but these jobs are run in the long worker suitable for long-running jobs.

  • all

    The all event is triggered every 60 seconds. This can be configured via the scheduler_tick_interval key in [common_site_config.json](https://frappeframework.com/docs/v13/user/en/basics/sites#scheduler_tick_interval)

  • cron

    A valid cron string that can be parsed by croniter.

Usage Examples:

scheduler_events = {
    "daily": [
        "app.scheduled_tasks.manage_recurring_invoices"
    ],
    "daily_long": [
        "app.scheduled_tasks.take_backups_daily"
    ],
    "cron": {
        "15 18 * * *": [
            "app.scheduled_tasks.delete_all_barcodes_for_users"
        ],
        "*/6 * * * *": [
            "app.scheduled_tasks.collect_error_snapshots"
        ],
        "annual": [
            "app.scheduled_tasks.collect_error_snapshots"
        ]
    }
}