django-minio-backend

The django-minio-backend provides a wrapper around the MinIO Python SDK. See minio/minio-py for the source.

Requirements & Compatibility

Django 4.2 or later
Python 3.11.0 or later
MinIO SDK 7.2.8 or later (installed automatically)

What's in the box?

The following set of features are available in django-minio-backend: * Django File Storage System Integration * Compliance with the django.core.files.storage.Storage class * Compliance with the STORAGES setting introduced in Django 4.2 * Static Files Support * Utilise and manage private and public buckets * Create buckets with custom policy hooks (MINIO_POLICY_HOOKS) * Consistency Check on Start (MINIO_CONSISTENCY_CHECK_ON_START) * Bucket Check on Upload (MINIO_BUCKET_CHECK_ON_SAVE) * Health Check (MinioBackend.is_minio_available()) * Docker Networking Support * Management Commands: * initialize_buckets * is_minio_available * clean_orphaned_minio_files * URL Caching for improved performance * Support for multiple MinIO backends via separate settings.py entries

For more about STORAGES, see Django 4.2 release notes / Custom file storages.

Migrating from v3.x to v4.0.0+

If your project is using an older version (<4.0.0) of django-minio-backend, see MIGRATIONS.md.

Integration

Get and install the package: bash pip install django-minio-backend
Add django_minio_backend.apps.DjangoMinioBackendConfig to INSTALLED_APPS: python INSTALLED_APPS = [ # '...' 'django_minio_backend.apps.DjangoMinioBackendConfig', # https://github.com/theriverman/django-minio-backend ]
Add the following parameters to your settings.py: ```python from datetime import timedelta

STORAGES = { # -- ADDED in Django 4.2 # "staticfiles": { # <-- DEFAULT STATIC FILES STORAGE # "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage", # Uncomment this storage to use Django's default static files storage # and add STATIC_ROOT outside the STORAGES dict # }, "staticfiles": { "BACKEND": "django_minio_backend.models.MinioBackendStatic", "OPTIONS": { "MINIO_ENDPOINT": "play.min.io", "MINIO_ACCESS_KEY": "Q3AM3UQ867SPQQA43P2F", "MINIO_SECRET_KEY": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", "MINIO_USE_HTTPS": True, "MINIO_REGION": "us-east-1", "MINIO_URL_EXPIRY_HOURS": timedelta(days=1), # Default is 7 days (longest) if not defined "MINIO_CONSISTENCY_CHECK_ON_START": True, "MINIO_STATIC_FILES_BUCKET": "my-static-files-bucket", }, }, "default": { "BACKEND": "django_minio_backend.models.MinioBackend", "OPTIONS": { "MINIO_ENDPOINT": "play.min.io", "MINIO_EXTERNAL_ENDPOINT": "external.min.io", "MINIO_EXTERNAL_ENDPOINT_USE_HTTPS": True, "MINIO_ACCESS_KEY": "Q3AM3UQ867SPQQA43P2F", "MINIO_SECRET_KEY": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", "MINIO_USE_HTTPS": True, "MINIO_REGION": "us-east-1", "MINIO_PRIVATE_BUCKETS": ['django-backend-dev-private', 'my-media-files-bucket', ], "MINIO_PUBLIC_BUCKETS": ['django-backend-dev-public', 't5p2g08k31', '7xi7lx9rjh' ], "MINIO_URL_EXPIRY_HOURS": timedelta(days=1), # Default is 7 days (longest) if not defined "MINIO_CONSISTENCY_CHECK_ON_START": False, "MINIO_POLICY_HOOKS": [ # List[Tuple[str, dict]] # ('django-backend-dev-private', dummy_policy) ], "MINIO_DEFAULT_BUCKET": "my-media-files-bucket", "MINIO_STATIC_FILES_BUCKET": "my-static-files-bucket", "MINIO_BUCKET_CHECK_ON_SAVE": False, # (OPTIONAL) MULTIPART UPLOAD "MINIO_MULTIPART_UPLOAD": False, # False by default "MINIO_MULTIPART_THRESHOLD": 10 * 1024 * 1024, # 10MB default "MINIO_MULTIPART_PART_SIZE": 10 * 1024 * 1024, # 10MB default # (OPTIONAL) URL CACHING "MINIO_URL_CACHING_ENABLED": True, # Enable URL caching (disabled by default) "MINIO_URL_CACHE_TIMEOUT": 60 * 60 * 8, # 8 hours in seconds "MINIO_URL_CACHE_PREFIX": 'minio_url_', # Prefix for cache keys }, }, }

Custom HTTP Client (OPTIONAL)

import os import certifi import urllib3 timeout = timedelta(minutes=5).seconds ca_certs = os.environ.get('SSL_CERT_FILE') or certifi.where() STORAGES["default"]["OPTIONS"]["MINIO_HTTP_CLIENT"]: urllib3.poolmanager.PoolManager STORAGES["default"]["OPTIONS"]["MINIO_HTTP_CLIENT"] = urllib3.PoolManager( timeout=urllib3.util.Timeout(connect=timeout, read=timeout), maxsize=10, cert_reqs='CERT_REQUIRED', ca_certs=ca_certs, retries=urllib3.Retry( total=5, backoff_factor=0.2, status_forcelist=[500, 502, 503, 504] ) ) ```
Implement your own lazy-loaded Attachment handler and integrate django-minio-backend: ```python # storages.py from django_minio_backend.models import MinioBackend

def get_public_storage(): return MinioBackend( bucket_name='django-backend-dev-public', storage_name='default', )

def get_private_storage(): return MinioBackend( bucket_name='django-backend-dev-private', storage_name='default', ) ```

```python

models.py

from django.db import models from django_minio_backend import MinioBackend, iso_date_prefix from .storages import get_public_storage, get_private_storage

class PrivateAttachment(models.Model):
file: FieldFile = models.FileField(verbose_name="Object Upload", storage=get_private_storage, upload_to=set_file_path_name) `` **Note:** It is highly recommended to declare yourMinioBackend` class instances in a callable function to avoid future Django migration problems due to Django's model serialisation.
Initialise the buckets & set their public policy (OPTIONAL):
This django-admin command creates both the private and public buckets in case one of them does not exist, and sets the public bucket's privacy policy from private(default) to public.
bash python manage.py initialize_buckets

Code reference: initialize_buckets.py.

Static Files Support

django-minio-backend allows serving static files from MinIO. To learn more about Django static files, see Managing static files, STATICFILES_STORAGE and STORAGES.

To enable static files support, update your settings.py:

STORAGES = {  # -- ADDED IN Django 5.1
    "default": {
        "BACKEND": "django_minio_backend.models.MinioBackend",
    },
    "staticfiles": {  # -- ADD THESE LINES FOR STATIC FILES SUPPORT
        "BACKEND": "django_minio_backend.models.MinioBackendStatic",
        "OPTIONS": {
            "MINIO_ENDPOINT": "play.min.io",  # ONLY EXTERNAL ADDRESS FOR STATIC
            "MINIO_ACCESS_KEY": "Q3AM3UQ867SPQQA43P2F",
            "MINIO_SECRET_KEY": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
            "MINIO_USE_HTTPS": False,
            "MINIO_REGION": "us-east-1",
            "MINIO_URL_EXPIRY_HOURS": timedelta(days=1),  # Default is 7 days (longest) if not defined
            "MINIO_CONSISTENCY_CHECK_ON_START": True,
            "MINIO_STATIC_FILES_BUCKET": "my-static-files-bucket",
        },
    },
}

The value of STATIC_URL is ignored, but it must be defined otherwise Django will throw an error.

IMPORTANT
The bucket configured in MINIO_STATIC_FILES_BUCKET is forced to be public to avoid known Django errors. It is not supported to serve Django static files from a private bucket.

Note: If MINIO_STATIC_FILES_BUCKET is not set, the default value (auto-generated-bucket-static-files) will be used.

Default File Storage Support

django-minio-backend can be configured as a default file storage. To learn more, see STORAGES.

To configure django-minio-backend as the default file storage, update settings.py with MINIO_DEFAULT_BUCKET:

STORAGES = {  # -- ADDED IN Django 5.1
    "default": {
        "BACKEND": "django_minio_backend.models.MinioBackend",
        "OPTIONS": {
            # ...
            "MINIO_DEFAULT_BUCKET": "django-minio-backend-default-dev-bucket",  # PRIVATE by default if not declared below
            "MINIO_PRIVATE_BUCKETS": [ ],
            "MINIO_PUBLIC_BUCKETS": ['django-minio-backend-default-dev-bucket'],
            # ...
        },
    },
}

IMPORTANT
The value set in MINIO_DEFAULT_BUCKET can be added either to MINIO_PRIVATE_BUCKETS or MINIO_PUBLIC_BUCKETS. By default, django-minio-backend will register MINIO_DEFAULT_BUCKET as private.

Note: If MINIO_DEFAULT_BUCKET is not set, the default value (auto-generated-bucket-media-files) will be used. Policy setting for default buckets is private.

URL Caching

django-minio-backend includes a caching layer for pre-signed URLs to improve performance. This is particularly useful for applications that frequently access the same files, as it prevents regenerating URLs for the same objects repeatedly.

The cache uses Django's cache framework and stores URLs with a key based on the bucket name, file name, and ETag for uniqueness. For public buckets, caching is skipped as the URLs are static.

URL caching is disabled by default. To enable and configure URL caching, you can set the following parameters in your settings.py:

# URL caching configuration (optional)
MINIO_URL_CACHING_ENABLED = True  # Enable URL caching (disabled by default)
MINIO_URL_CACHE_TIMEOUT = 60 * 60 * 8  # 8 hours in seconds
MINIO_URL_CACHE_PREFIX = 'minio_url_'  # Prefix for cache keys

If MINIO_URL_CACHE_TIMEOUT is not set, it defaults to 80% of MINIO_URL_EXPIRY_HOURS to ensure cached URLs don't expire before they're regenerated.

Health Check

To check the connection link between Django and MinIO, use the provided MinioBackend.is_minio_available() method.
It returns a MinioServerStatus instance which can be quickly evaluated as boolean.

Example:

from django_minio_backend import MinioBackend

minio_available = MinioBackend().is_minio_available()  # The bucket_name parameter can be empty for this check
if minio_available:
    print("OK")
else:
    print("NOK")
    print(minio_available.details)

Policy Hooks

You can configure django-minio-backend to automatically execute a set of pre-defined policy hooks.
Policy hooks can be defined in settings.py by adding MINIO_POLICY_HOOKS which must be a list of tuples.
Policy hooks are automatically picked up by the initialize_buckets management command.

For an exemplary policy, see the implementation of set_bucket_to_public() in django_minio_backend/models.py or the contents of examples/policy_hook.example.py.

Consistency Check On Start

When enabled, the initialize_buckets management command gets called automatically when Django starts.
This command connects to the configured MinIO server and checks if all buckets defined in settings.py are present.
In case a bucket is missing or its configuration differs, it gets created and corrected.

Management Commands

initialize_buckets

This django-admin command creates both the private and public buckets in case one of them does not exist, and sets the public bucket's privacy policy from private(default) to public.
bash python manage.py initialize_buckets

Code reference: initialize_buckets.py.

clean_orphaned_minio_files

This management command helps maintain the integrity of your MinIO storage by:

Identifying and removing orphaned files in MinIO buckets that are no longer referenced in the database
Identifying database references to files that no longer exist in MinIO buckets

Usage:

python manage.py clean_orphaned_minio_files [--dry-run] [--check-missing]

Options: - --dry-run: Run without actually deleting files (simulation mode) - --check-missing: Check for database references to missing MinIO files

This command is useful for periodic maintenance to ensure your storage and database remain in sync, preventing storage leaks and identifying broken references.

Note: The on-start consistency check equals to manually calling python manage.py initialize_buckets.
It is recommended to turn off this feature during development by setting MINIO_CONSISTENCY_CHECK_ON_START to False, because this operation can noticeably slow down Django's boot time when many buckets are configured.

Reference Implementation

For a reference implementation, see Examples.

Behaviour

The following list summarises the key characteristics of django-minio-backend: * The STORAGES setting introduced in Django 4.2 enables configuring multiple storage backends with great customisation. * MEDIA and STATIC files must be configured separately and the latter must be named staticfiles explicitly. * STATIC files are stored in a single bucket managed via MINIO_STATIC_FILES_BUCKET. * STATIC files are public by default and must remain public to avoid certain Django admin errors. * The value of MEDIA_URL is ignored, but it must be defined otherwise Django will throw an error. * If you're serving static files with the default backend, add STATIC_ROOT outside STORAGES in your settings. * A bucket's existence is not checked on a save by default. To enable this guard, set MINIO_BUCKET_CHECK_ON_SAVE = True in your settings.py. * Bucket existence is not checked on Django start by default. To enable this guard, set MINIO_CONSISTENCY_CHECK_ON_START = True in your settings.py. * Many configuration errors are validated through AppConfig but not every error can be captured there. * Files with the same name in the same bucket are not replaced on save by default. Django will store the newer file with an altered file name. To allow replacing existing files, pass the replace_existing=True kwarg to MinioBackend. For example: python image = models.ImageField(storage=storages.get_private_images_storage()) * Depending on your configuration, django-minio-backend may communicate over two kinds of interfaces: internal and external. If your settings.py defines a different value for MINIO_ENDPOINT and MINIO_EXTERNAL_ENDPOINT, then the former will be used for internal communication between Django and MinIO, and the latter for generating URLs for users. This behaviour optimises the network communication. See Networking below for a thorough explanation. * The uploaded object's content-type is guessed during save. If mimetypes.guess_type fails to determine the correct content-type, then it falls back to application/octet-stream.

Networking and Docker

If your Django application is running on a shared host with your MinIO instance, you should consider using the MINIO_EXTERNAL_ENDPOINT and MINIO_EXTERNAL_ENDPOINT_USE_HTTPS parameters. This way most traffic will happen internally between Django and MinIO. The external endpoint parameters are required for external pre-signed URL generation.

If your Django application and MinIO instance are running on different hosts, you can omit the MINIO_EXTERNAL_ENDPOINT and MINIO_EXTERNAL_ENDPOINT_USE_HTTPS parameters, and django-minio-backend will default to the value of MINIO_ENDPOINT.

Setting up and configuring custom networks in Docker is not in the scope of this document.
To learn more about Docker networking, see Networking overview and Networking in Compose.

See CONTRIBUTE.Docker.md for a real-life Docker Compose demonstration.

Contribution

Please find the details in CONTRIBUTE.md and CONTRIBUTE.Docker.md.

Copyright

theriverman/django-minio-backend licensed under the MIT License
minio/minio-py is licensed under the Apache License 2.0