django-huey-monitor
Django based tool for monitoring huey task queue
Current implementation will just store all Huey task signals into the database and display them in the Django admin.
Quickstart
pip install django-huey-monitor
INSTALLED_APPS = [
#...
'bx_django_utils', # https://github.com/boxine/bx_django_utils
'huey_monitor',
#...
]
Note: You must normally not change your Huey tasks.
Collect main-/sub-tasks
Huey monitor model can store information about task hierarchy. But this information can't be collected automatically. You have to store these information in your Task code.
e.g.:
@task(context=True)
def sub_task(task, parent_task_id, chunk_data):
# Save the task hierarchy by:
TaskModel.objects.set_parent_task(
main_task_id=parent_task_id,
sub_task_id=task.id,
)
# ... do something with e.g.: chunk_data ...
@task(context=True)
def main_task(task):
for chunk_data in something:
sub_task(parent_task_id=task.id, chunk_data=chunk_data)
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
Collect progress information
Running task can store progress information in a similar way as tqdm. So it's possible to see the progress in admin.
Minimal example:
@task(context=True)
def foobar_task(task, list_to_process):
process_info = ProcessInfo(
task,
desc='A description of this Job',
total=len(list_to_process)
)
for item in list_to_process:
# ...to something with the {item}...
process_info.update(n=1) # add the information that one item was processed
It is also possible to divide the work to several tasks and collect information about the processing of main-/sub-tasks.
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
settings
override list filter (optional)
It is possible to override list_filter
of SignalInfoModelAdmin
and TaskModelAdmin
via settings.
e.g.:
HUEY_MONITOR_SIGNAL_INFO_MODEL_LIST_FILTER = ('task__name', 'signal_name')
HUEY_MONITOR_TASK_MODEL_LIST_FILTER = ('name', 'state__signal_name')
Note: This both settings are optional. In this example is the "hostname" filter not present ;)
run test project
Note: You can quickly test Huey Monitor with the test project, e.g:
~/django-huey-monitor$ ./manage.sh run_testserver
or in an isolated Docker container:
~/django-huey-monitor$ make up
More info see below.
Screenshots
(More Screenshots here: boxine.github.io/django-huey-monitor/)
2021-02-22-v030-task-details.png
2021-02-22-v030-progress-info1.png
developing
- install docker
- clone the project
- start the container
To start developing e.g.:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py
~/django-huey-monitor$ make help
~/django-huey-monitor$ make up
Point our browser to: http://localhost:8000/
Our Makefile contains the following targets:
help List all commands
install install huey monitor package
update Update the dependencies as according to the pyproject.toml file
run-dev-server Run Django's developer server
test Run unittests
tox Run unittests via tox
makemessages Make and compile locales message files
clean Remove created files from the test project (e.g.: SQlite, static files)
build Update/Build docker services
up Start docker containers
down Stop all containers
shell-django go into a interactive bash shell in Django container
run-shell-django Build and start the Django container and go into shell
shell-huey1 go into a interactive bash shell in Huey worker container 1
shell-huey2 go into a interactive bash shell in Huey worker container 2
shell-huey3 go into a interactive bash shell in Huey worker container 3
shell-redis go into a interactive bash shell in Redis container
logs Display and follow docker logs
logs-django Display and follow docker logs only from "django" container
reload-django Reload the Django dev server
reload-huey Reload the Huey worker
restart Restart the containers
fire-test-tasks Call "fire-test-tasks" manage command to create some Huey Tasks
fire-many-test-tasks Call "fire-test-tasks" with --count 10000 to create many task entries ;)
fire-parallel-processing-task Just fire "parallel processing" Huey Task
delete-all-tasks-data Delete all Task/Signal database enties
It's also possible to run the test setup with SQLite and Huey immediate setup without docker:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py run_dev_server
Backwards-incompatible changes
Version compatibility
Huey Monitor | Django | Python |
---|---|---|
>v0.7.0 | v3.2, v4.1, v4.2 | v3.9, v3.10, v3.11 |
>v0.6.0 | v3.2, v4.0, v4.1 | v3.9, v3.10, v3.11 |
>v0.5.0 | v2.2, v3.1, v3.2 | v3.7, v3.8, v3.9 |
<=v0.4.0 | v2.2, v3.0, v3.1 | v3.7, v3.8, v3.9 |
v0.6.0
We refactor the project setup: Developer must reinit the repository.
v0.5.0
Change CI and remove tests against Django 3.0, but add test run with Django v3.2
v0.3.0 -> v0.4.0 - Outsourcing Django stuff
bx_py_utils was split and Django related stuff moved into: bx_django_utils
You must change your Django settings and replace the app name:
INSTALLED_APPS = [
#...
- 'bx_py_utils',
+ 'bx_django_utils',
'huey_monitor',
#...
]
History
- dev
- tbc
- v0.9.1 - 26.01.2024
- Fix
DisallowedModelAdminLookup
inSignalInfoModelAdmin
, too. - v0.9.0 - 22.12.2023
- Fix #135
DisallowedModelAdminLookup
inTaskModelAdmin
- Add "thread" name as change list filter.
- Enhance test project setup
- Apply manageprojects updates
- Remove Python v3.9 support
- Add Django v5.0 to test matrix and remove Django 4.1
- Enable local AUTOLOGIN as default
- Use unittest "load_tests Protocol" and deny any requests in tests
- Add https://github.com/PyCQA/flake8-bugbear
- Update requirements
- v0.8.1 - 20.11.2023
- Bugfix
ZeroDivisionError
in admin - v0.8.0 - 20.11.2023
- Make is possible to override
list_filter
ofSignalInfoModelAdmin
andTaskModelAdmin
via settings - Update local docker dev setup
- v0.7.1 - 18.08.2023
- Fix #127: Catch error getting HUEY counts
- v0.7.0 - 09.08.2023
- New: Display Huey pending/scheduled/result task counts in admin
- Switch to git
main
branch - Switch from
pytest
to normalunittests
- Switch from
poetry
topip-tools
- Use https://github.com/jedie/manage_django_project for developing
- Expand test matrix by remove Django 4.0 and add 4.2
- Enhance tox config
- v0.6.0 - 09.01.2023
- Test against Django v3.2, v4.0, v4.1 and Python v3.9 - v3.11
- Optimize Admin change list (contributed by henribru)
- Order sub-tasks chronologically in admin (contributed by Skrattoune)
- displaying progression for parent_tasks (contributed by Skrattoune)
- Delegating set_parent_task to
ProcessInfo.__init__
(contributed by Skrattoune) - Fix #27 auto crop overlong process info description
- NEW #102 Unlock all tasks via change list object tool link
- Fix #81 Expand
TaskModel.desc
max. length and makeSignalInfoModel.exception_line
optional - Update docker test setup
- v0.5.0 - 10.02.2022
- Refactor models: Remove
TaskProgressModel
and store progress information intoTaskModel
- v0.4.6 - 03.02.2022
- Display task desc or name in TaskAdmin (contributed by Skrattoune)
- update requirements
- use Darker as code formatter
- Check linting via PyTest plugins
- v0.4.5 - 28.01.2022
- Fix #46 by increment existing TaskProgressModel instances (based on contribution by Skrattoune)
- v0.4.4 - 07.01.2022
- Add missing
huey_monitor.css
file - v0.4.3 - 07.01.2022
- Add temporary
cumulate2parents
flag toProcessInfo
(contributed by formacube) - v0.4.2 - 25.08.2020
- suppress the default_app_config attribute in Django 3.2+ (contributed by Jonas Svarvaa)
- v0.4.1 - 02.06.2020
- Bugfix
ProcessInfo.__str__()
- #27 Check 'desc' length in
ProcessInfo
- Remove test against Django 3.0 and add tests with Django 3.2
- Bugfix local
tox
runs and use different Python versions - v0.4.0 - 21.05.2020
- bx_py_utils was split and Django related stuff moved into: https://github.com/boxine/bx_django_utils
- v0.3.0 - 22.02.2020
- Store and display running task progress information a little bit as tqdm #17
- v0.2.0 - 17.02.2020
- Store "parent_task" information for main-/sub-tasks
- v0.1.0 - 21.12.2020
- Work-a-round if a Huey worker died
- Fix missing translations from bx_py_utils in test project
- Simulate a cluster of Huey worker via docker and the test project
- v0.0.1 - 15.12.2021
- initial release
License
GPL. Patches welcome!
Links
- https://pypi.org/project/django-huey-monitor/