What is Recruiter¶
Features and characteristics:
- Jobs are made persistent on MongoDB
- Jobs with complex and customizable strategies
- Jobs are stored by default in an history collection for after the fact inspection and analytics
- Multiple queues are supported through grouping
- Built to be robust, scalable and fault tolerant
At high level, it provides a few major components:
- A recruiter: a single instance long-running process who assign enqueued jobs to a worker
- A worker: a multiple instace long-running processes that each execute a single job at a time
- A cleaner: a single instance long-running process who takes care of cleaning up the dirty conditions that can happen (i.e. worker dead for a fatal error)
Why¶
DA COMPLETARE¶
Onebip is a payment system (think PayPal with mobile devices in place of credit cards), things like: payment notifications, subscription renewals, remainder messages, … are really important. You cannot skip or lose a job (notification are idempotent but payments are not). You cannot forgot to have completed a job (customer/merchant support must have data to do their job). You need to know if and when you can retry a failed job (external services have rate limits and are based on agreements/contracts). We have developed internally our job/queue solution called Recruiter. After a year in production and many billions of jobs we have decided to put what we have learned into a stand alone project and to make it available to everyone.