Celery Alternative for Django

Background

So today I will be talking about a Celery alternative named Huey, which comes with a much easier setup than Celery and is much smaller in size compared to Celery.

The reason why I decided to try out Huey is because I have faced some issues with Celery sometimes for doing some common tasks because the documentation isn't too great.

For those who don't know what Celery is or haven't used it before, Huey is an asynchronous task queue which allows you to perform scheduled tasks or long running tasks in the background.

Prerequisites

We will be installing the following packages:

redis
django
huey
requests (optional, needed for the demo)

Github Repo

The following blog comes accompanied with a github repo, that you can use to test out the demo project that we will be creating.

Click here to view repo.

Project Setup

Create Project Directory

Open the terminal and type the following to create a directory, you can skip this step and do it from File Explorer itself.

mkdir huey_demo

Virtual Environment

Let us create a virtualenv first to install our project dependencies:
```
python -m venv venv
```
Activate the virtualenv (Linux):
```
source venv/bin/activate
```

Installing dependencies

Type the following command in the terminal to install all the dependencies:

pip install Django==4.0.4 redis==4.2.2 huey==2.4.3

At the time of writing this article, these were the versions I tested out this setup with, keep an eye on the Github Repo for any updates as per the latest version in the future.

Create Project

Create the django project by typing the following command in terminal:
```
django-admin startproject django_huey_demo
```
Change directory into django project directory:
```
cd django_huey_demo
```
Create the app under our project:
```
python manage.py startapp demo
```

Include the created app into project settings.py, make the following changes:

INSTALLED_APPS = [
    # Existing Apps
    "demo.apps.DemoConfig",  # <== Add this line
]

Set debug mode to False in settings.py:
```
DEBUG=False
```
We are setting Debug to False so that we can see how Huey runs in production, more on this later.

Project Overview

Now that we are done setting up our project, it is a good time to take you over what we will be building today. We will fetch "Word of the Day" daily from Wordnik API. Then we will store the word, its definition, and an example of the word in a sentence to our database.

We will setup a periodic task using Huey which will fetch the Word of the Day and store it.

For storing the word we will be creating a Django Model of the same.

Getting Wordnik API Key

You can follow this guide to obtain the API key.

Coding our project

Add Huey to our Project

We need to add Huey to installed apps of our project, so make the following changes in settings.py file:

INSTALLED_APPS = [
    # Existing apps
    'huey.contrib.djhuey', # <== Add this line
]

Install Redis

We need to install Redis for Huey to store information about queued tasks in it, like we used to do with Celery as well. You can refer to the following link to install redis based on your specific operating system.

If you're comfortable with using Docker, you can use the following command:

docker run --name redis_huey -p 6379:6379 -d redis

By default, Huey will try connecting to Redis server running on localhost:6379. If it isn't present, it will raise an error.

Model Definition

Add the following code to your demo/models.py file:

from django.db import models

class Word(models.Model):
    word = models.CharField(max_length=200)
    part_of_speech = models.CharField(max_length=100)
    definition = models.TextField()
    example = models.TextField()

    def __str__(self):
        return self.word

Make migrations:
```
python manage.py makemigrations demo
```
Apply migrations:
```
python manage.py migrate demo
```

Task Definition

Create a file named tasks.py in demo app directory. The reason why we named our file tasks.py is to help Huey auto discover the tasks present in our registered apps, if we named our file anything other than that, we would have to manually register our task. If you would like to know more you can check out the Huey documentation here.

Before we write the task definition we need to install an additional dependency requests. Install it by typing the following in your terminal:

pip install requests==2.27.1

Now comes the code:

import requests
from django.conf import settings
from huey import crontab
from huey.contrib.djhuey import db_periodic_task

from demo.models import Word


@db_periodic_task(crontab(hour="18", minute="00"))
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    data = r.json()
    Word.objects.get_or_create(
        word=data["word"],
        part_of_speech=data["definitions"][0]["partOfSpeech"],
        definition=data["definitions"][0]["text"],
        example=data["examples"][0]["text"]
    )

Add the following line in your project settings:

WORDNIK_API_KEY = "api-key-here"

This codeblock might be a lot to take in, so let's go over the things in it one by one:

Huey Decorator
```
from huey.contrib.djhuey import db_periodic_task
```
This is a decorator provided by Huey to register periodic task that involve working with database, this decorator automatically closes the database connection upon task completion, for more details, you can refer here.
Crontab Schedule
```
@db_periodic_task(crontab(hour="18", minute="00"))
```
We are passing the argument crontab(hour="18", minute="00") to our periodic task decorator, this tells Huey to run our task at 6pm everyday. You can make use of this website to create your crontab schedules, I use it everytime.
Wordnik API Key
```
from django.conf import settings

# Usage
## settings.WORDNIK_API_KEY
```
from django.conf import settings is the standard way to import any data from our project settings, it is useful in cases where we have multiple settings files setup for different environment so it will know which file to pick from without us having to worry about it. It finds out which settings file we are using from the DJANGO_SETTINGS_MODULE environment variable. But you don't have to worry about these details.

Then we are using the key in our wordnik API call.
Wordnik API Call
```
r = requests.get(
f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
```
Here we are making use of the requests module to make a GET request to the wordnik API while passing our API Key for authentication.
Storing word in database
```
data = r.json()
Word.objects.get_or_create(
    word=data["word"],
    part_of_speech=data["definitions"][0]["partOfSpeech"],
    definition=data["definitions"][0]["text"],
    example=data["examples"][0]["text"]
)
```
After parsing the API response we are storing the word definition in our database. We are making use of get_or_create method instead of create method here so that we don't create multiple copies of the same word in our database if that word is ever repeated by the Wordnik API.

Wordnik API Response
Here is what the Wordnik API response for Word of the Day endpoint looks like. Some of the irrelevant sections of the response has been truncated for brevity purposes.

{
"word": "stolon",
"definitions": [
    {
    "source": "ahd-5",
    "text": "A long thin stem that usually grows horizontally along the ground and produces roots and shoots at widely spaced nodes, as in a strawberry plant.",
    "note": null,
    "partOfSpeech": "noun"
    },
    // More definitions here...
],
"publishDate": "2022-05-08T03:00:00.000Z",
"examples": [
    {
    "title": "4.1 Nursery establishment",
    "text": "A stolon is a stem that grows along the ground, producing at its nodes new plants with roots and upright stems.",
    // Additional data here...
    },
    // More examples here...
],
    // Additional fields here...
}

Running Huey Worker

You can start the Huey worker by typing the following command in your terminal:

python manage.py run_huey

You can pass multiple flags to the above command which will change what gets logged to the console, such as:

-v, --verbose - verbose logging (includes DEBUG level)
-q, --quiet - minimal logging
-S, --simple - simple logging format (“time message”)

To look at various other options for logging, checkout the docs here.

What else can you do with Huey?

Task Decorators

Huey comes with multiple task decorators depending on what operations you are performing within the task. I'll explain in brief what all of those do below. Here is the import statement for all the decorators:

from huey.contrib.djhuey import task, periodic_task, db_task, db_periodic_task

task: A regular task
periodic_task : When you want to run a task periodically based on a schedule
db_task : When you want to perform db operations within your task
db_periodic_task : When you want to perform db operations in a periodic task

Crontab Examples

Let me show you some more examples of how you can use crontab to schedule your tasks.

crontab(minute='*/3') would schedule the task to run every three minute
crontab(hour='*/3', minute='5') would create a task that will run at 5 minute past every third hour.
crontab(minute='00', hour='10', month='*/2', day_of_week='*/5') would create a task that would run on every 5th day of the week, of every 2nd month at 10:00 AM.

Scheduling Tasks

For example, you have the following task defined inside tasks.py:

from huey.contrib.djhuey import task

@task()
def count():
    for i in range(10):
        print(i)

Now you want to call this task, but want it to run after 5 seconds, you can do the following:

count.schedule(delay=5)

delay parameter takes values in seconds, so if you want it to execute after 5 minutes specify 300 seconds.

Retrying tasks that fail

Suppose you add the following logic to our existing task:

@db_periodic_task(crontab(hour="18", minute="00"), retries=2)
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    if r.status_code != 200:
        raise Exception("Unable to fetch data from Wordnik API")  ## Add this logic
    else:
        data = r.json()
        Word.objects.get_or_create(
            word=data["word"],
            part_of_speech=data["definitions"][0]["partOfSpeech"],
            definition=data["definitions"][0]["text"],
            example=data["examples"][0]["text"]
        )

So we added the logic to check for status code of the response, and if it is something other than 200, it will retry that task upto 2 times. But these retries would happen without any time gap between the two attempts. Now what if you want to delay out multiple attempts of this task, we can do that by passing the retry_delay argument, it accepts values in seconds.

@db_periodic_task(crontab(hour="18", minute="00"), retries=2, retry_delay=10)

This will cause a 10 second delay between multiple attempts.

Development Mode

Huey comes with a default setting which makes working with Huey during development in Django easier. So whenever you have DEBUG=True present in your settings.py file, tasks will be executed synchronously just like regular function calls. The purpose of this is to avoid running both Redis and an additional consumer process while developing or running tests. You can read more about this here.

For this, we need to add the following line in settings.py:

HUEY = {}

However, if you want to override this behaviour, you can add the following Huey config instead:

HUEY = {
    "immediate": False
}

If you have the above config mentioned in settings.py while having DEBUG=True Huey will require you to setup Redis and run huey worker using run_huey command.

Celery vs Huey

Some observations about Huey as compared to Celery are:

Smaller footprint of dependencies as compared to Celery. Celery installs kombu and billiards along with it. Meanwhile, huey doesn't have any dependencies.
Lesser services needed to be run for periodic tasks, Celery requires running beat service and a worker service to work with periodic task meanwhile we only need to run one service using the run_huey command.

Celery Alternative for Django - Huey