Introducti on to Celery
What is Celery? “Celery is an asynchronous task queue based on distributed message passing.” It is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background.
Why should I use Celery?
User perspective • Minimize request/response cycle • Smoother user experience • Difference between pleasant and unpleasant
Developer perspective • Offload time/cpu intensive processes • Scalability - add workers as needed • Flexibility - many points of customization • Actively developed • Great documentation • Lots of tutorials
OUT OF THE REQUEST/RESPONSE CYCLE • Example: Sending emails asynchronously.
TASKS IN THE BACKGROUND. • Example: Computational heavy jobs. • Example: Interacting with external APIs.
PERIODIC JOBS
Celery Architecture
PRODUCER • Produces a task for the queue.
BROKER • Stores the task backlog • Answers, what work remains to be done? • RabbitMQ, Redis, SQLAlchemy, Django's ORM, MongoDB
WORKER • Execute and consumes tasks. • Distributed.
RESULTS BACKEND • Stores the results from our tasks.
EXAMPLE
INTERGRATING WITH DJANGO.
BEWARE OF DJANGO- CELERY
IMPORTANT SETTINGS • BROKER_URL = 'redis://localhost:6379/0' • CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' • CELERY_TASK_RESULT_EXPIRES = 7*86400 # 7 days • CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler" • CELERY_ACCEPT_CONTENT = ['application/json'] • CELERY_TASK_SERIALIZER = 'json' • CELERY_RESULT_SERIALIZER = 'json' • BROKER_BACKEND = "redis" • BROKER_CONNECTION_TIMEOUT = 5.0 • CELERY_IMPORTS = ('<your-app-name>.tasks',)
Write code in celery.py
Intialize celery in __init__.py
BEST PRACTICES
NEVER PASS OBJECTS AS ARGUMENTS.
AVOID LAUNCHING SYNCHRONOUS SUBTASKS
PERIODIC TASKS
THINGS GO WRONG ?
RE-TRY!
Thanks!

Introduction to Django-Celery and Supervisor