Merge branch '862-uvicorn' into 'develop'

Fix #862: replaced Daphne by Gunicorn/Uvicorn

Closes #862

See merge request funkwhale/funkwhale!790

.gitlab-ci.yml

@@ -129,6 +129,7 @@ test_api:
   only:
     - branches
   before_script:
+    - apk add make
     - cd api
     - sed -i '/Pillow/d' requirements/base.txt
     - pip3 install -r requirements/base.txt

api/Dockerfile

@@ -44,7 +44,7 @@ RUN \
     if [ "$install_dev_deps" = "1" ] ; then echo "Installing dev dependencies" && pip3 install --no-cache-dir -r /requirements/local.txt -r /requirements/test.txt ; else echo "Skipping dev deps installation" ; fi
 
 ENTRYPOINT ["./compose/django/entrypoint.sh"]
-CMD ["./compose/django/daphne.sh"]
+CMD ["./compose/django/server.sh"]
 
 COPY . /app
 WORKDIR /app

api/compose/django/daphne.sh

@@ -1,3 +0,0 @@
-#!/bin/bash -eux
-python /app/manage.py collectstatic --noinput
-daphne -b 0.0.0.0 -p 5000 config.asgi:application --proxy-headers

api/compose/django/server.sh

@@ -0,0 +1,3 @@
+#!/bin/bash -eux
+python /app/manage.py collectstatic --noinput
+gunicorn config.asgi:application -w ${FUNKWHALE_WEB_WORKERS-1} -k uvicorn.workers.UvicornWorker -b 0.0.0.0:5000

api/requirements.apt

@@ -8,3 +8,4 @@ postgresql-client
 python3-dev
 libldap2-dev
 libsasl2-dev
+make

api/requirements/base.txt

@@ -50,6 +50,8 @@ python-magic==0.4.15
 channels==2.1.6
 channels_redis>=2.3,<2.4
 daphne>=2.2,<2.3
+uvicorn
+gunicorn
 
 cryptography>=2,<3
 # requests-http-signature==0.0.3

changes/changelog.d/862.enhancement

@@ -0,0 +1 @@
+Replaced Daphne by Gunicorn/Uvicorn to improve stability, flexibility and performance (#862)

changes/notes.rst

@@ -6,106 +6,22 @@ Next release notes
     Those release notes refer to the current development branch and are reset
     after each release.
 
-Edits on tracks, albums and artists
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Funkwhale was a bit annoying when it camed to metadata. Tracks, albums and artists profiles
-were created from audio file tags, but basically immutable after that (unless you had
-admin access to Django's UI, which wasn't ideal to do this kind of changes).
+Replaced Daphne by Gunicorn/Uvicorn [manual action required, non-docker only]
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-With this release, everyone can suggest changes on track, album and artist pages. Users
-with the "library" permission can review suggested edits in a dedicated interface
-and apply/reject them.
+To improve the performance, stability and reliability of Funkwhale's web processes,
+we now recommend using Gunicorn and Uvicorn instead of Daphne. This combination unlock new use cases such as:
 
-Approved edits are broadcasted via federation, to ensure other instances get the information
-too.
+- zero-downtime upgrades
+- configurable number of web worker processes
 
-Not all fields are currently modifiable using this feature. Especially, it's not possible
-to suggest a new album cover, or reassign a track to a different album or artist. Those will
-be implemented in a future release.
+Based on our benchmarks, Gunicorn/Unicorn is also faster and more stable under higher workloads compared to Daphne.
 
-Admin UI for tracks, albums, artists, libraries and uploads
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+To benefit from this enhancement on existing instances, you need to add ``FUNKWHALE_WEB_WORKERS=1`` in your ``.env`` file
+(use a higher number if you want to have more web worker processes).
 
-As part of our ongoing effort to make Funkwhale easier to manage for instance owners,
-this release includes a brand new administration interface to deal with:
+Then, edit your ``/etc/systemd/system/funkwhale-server.service`` and replace the ``ExecStart=`` line with
+``ExecStart=/srv/funkwhale/virtualenv/bin/gunicorn config.asgi:application -w ${FUNKWHALE_WEB_WORKERS} -k uvicorn.workers.UvicornWorker -b ${FUNKWHALE_API_IP}:${FUNKWHALE_API_PORT}``
 
-- tracks
-- albums
-- artists
-- libraries
-- uploads
-
-You can use this UI to quickly search for any object, delete objects in batch, understand
-where they are coming from etc. This new UI should remove the need to go through Django's
-admin in the vast majority of cases (but also includes a link to Django's admin when needed).
-
-
-Artist hiding in the interface
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-It's now possible for users to hide artists they don't want to see.
-
-Content linked to hidden artists will not show up in the interface anymore. Especially:
-
-- Hidden artists tracks are removed from the current queue
-- Starting a playlist will skip tracks from hidden artists
-- Recently favorited, recently listened and recently added widgets on the homepage won't include content from hidden artists
-- Radio suggestions will exclude tracks from hidden artists
-- Hidden artists won't appear in Subsonic apps
-
-Results linked to hidden artists will continue to show up in search results and their profile page remains accessible.
-
-OAuth2 authorization for better integration with third-party apps
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Funkwhale now support the OAuth2 authorization and authentication protocol which will allow
-third-party apps to interact with Funkwhale on behalf of users.
-
-This feature makes it possible to build third-party apps that have the same capabilities
-as Funkwhale's Web UI. The only exception at the moment is for actions that requires
-special permissions, such as modifying instance settings or moderation (but this will be
-enabled in a future release).
-
-If you want to start building an app on top of Funkwhale's API, please check-out
-`https://docs.funkwhale.audio/api.html`_ and `https://docs.funkwhale.audio/developers/authentication.html`_.
-
-Better error handling and display during import
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Funkwhale should now be more resilient to missing tags in imported files, and give
-you more insights when something goes wrong, including the specific tags that were missing
-or invalid, and additional debug information to share in your support requests.
-
-This information is available in all pages that list uploads, when clicking on the button next to the upload status.
-
-Support for S3-compatible storages to store media files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Storing all media files on the Funkwhale server itself may not be possible or desirable
-in all scenarios. You can now configure Funkwhale to store those files in a S3
-bucket instead.
-
-Check-out `https://docs.funkwhale.audio/admin/external-storages.html`_ if you want to use
-this feature.
-
-Prune library command
-^^^^^^^^^^^^^^^^^^^^^
-
-Users are often surprised by Funkwhale's tendency to keep track, album and artist
-metadata even if no associated files exist.
-
-To help with that, we now offer a ``prune_library`` management command you can run
-to purge your database from obsolete entries. `Please refer to our documentation
-for usage instructions <https://docs.funkwhale.audio/admin/commands.html#pruning-library>`_.
-
-Check in-place files command
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When using in-place import with a living audio library, you'll quite often rename or
-remove files from the file system. Unfortunately, Funkwhale keeps a reference to those
-files in the database, which results in unplayable tracks.
-
-To help with that, we now offer a ``check_inplace_files`` management command you can run
-to purge your database from obsolete files. `Please refer to our documentation
-for usage instructions <https://docs.funkwhale.audio/admin/commands.html#remove-obsolete-files-from-database>`_.
+Then reload the configuration change with ``sudo systemctl daemon-reload`` and ``sudo systemctl restart funkwhale-server``.

deploy/FreeBSD/funkwhale_server

@@ -25,8 +25,8 @@ funkwhale_server_user=funkwhale
 funkwhale_server_env=$(cat /usr/local/www/funkwhale/config/.env | grep -v ^# | xargs)
 command_interpreter="/usr/local/www/funkwhale/virtualenv/bin/python3"
 
-command="/usr/local/www/funkwhale/virtualenv/bin/daphne"
-command_args="-b 127.0.0.1 -p 5000 config.asgi:application --proxy-headers \
+command="/usr/local/www/funkwhale/virtualenv/bin/gunicorn"
+command_args="config.asgi:application -w 4 -k uvicorn.workers.UvicornWorker -b 127.0.0.1:5000 \
 >> /var/log/funkwhale/${name##funkwhale_}.log 2>&1 &"
 
 run_rc_command "$1"

deploy/Gentoo/funkwhale_server

@@ -3,9 +3,9 @@
 NAME=funkwhaleserver
 PIDFILE=/var/run/$NAME.pid
 USER=funkwhale
-DAEMON_ARGS="-b 127.0.0.1 -p 5000 config.asgi:application --proxy-headers "
-Daphne=/srv/funkwhale/virtualenv/bin/daphne
-WORKDIR=/srv/funkwhale/api 
+DAEMON_ARGS="config.asgi:application -w 4 -k uvicorn.workers.UvicornWorker -b 127.0.0.1:5000 "
+Gunicorn=/srv/funkwhale/virtualenv/bin/gunicorn
+WORKDIR=/srv/funkwhale/api
 
 depend() {
         need net redis postgresql nginx funkwhale_beat funkwhale_worker
@@ -16,7 +16,7 @@ start() {
         cd /srv/funkwhale/api
 	set -a && source /srv/funkwhale/config/.env && set +a
         echo 'Starting Funkwhale Server'
-	start-stop-daemon --start --user $USER --make-pidfile --pidfile $PIDFILE  -d $WORKDIR  --exec $Daphne -- $DAEMON_ARGS >> /var/log/funk/daphne.log 2>&1&
+	start-stop-daemon --start --user $USER --make-pidfile --pidfile $PIDFILE  -d $WORKDIR  --exec $Gunicorn -- $DAEMON_ARGS >> /var/log/funk/server.log 2>&1&
 	echo 'Funkwhale Server started'
 	echo
 	eend $?

deploy/env.prod.sample

@@ -34,7 +34,9 @@ FUNKWHALE_VERSION=latest
 # example: FUNKWHALE_API_PORT=5678
 FUNKWHALE_API_IP=127.0.0.1
 FUNKWHALE_API_PORT=5000
-
+# The number of web workers to start in parallel. Higher means you can handle
+# more concurrent requests, but also leads to higher CPU/Memory usage
+FUNKWHALE_WEB_WORKERS=1
 # Replace this by the definitive, public domain you will use for
 # your instance
 FUNKWHALE_HOSTNAME=yourdomain.funkwhale
@@ -168,4 +170,4 @@ AWS_STORAGE_BUCKET_NAME=
 # If you are using Amazon S3 to serve media directly, you will need to specify your region
 # name in order to access files. Example:
 #   AWS_S3_REGION_NAME=eu-west-2
-# AWS_S3_REGION_NAME= 
+# AWS_S3_REGION_NAME=

deploy/funkwhale-server.service

@@ -8,7 +8,6 @@ User=funkwhale
 # adapt this depending on the path of your funkwhale installation
 WorkingDirectory=/srv/funkwhale/api
 EnvironmentFile=/srv/funkwhale/config/.env
-ExecStart=/srv/funkwhale/virtualenv/bin/daphne -b ${FUNKWHALE_API_IP} -p ${FUNKWHALE_API_PORT} config.asgi:application --proxy-headers
-
+ExecStart=/srv/funkwhale/virtualenv/bin/gunicorn config.asgi:application -w ${FUNKWHALE_WEB_WORKERS} -k uvicorn.workers.UvicornWorker -b ${FUNKWHALE_API_IP}:${FUNKWHALE_API_PORT}
 [Install]
 WantedBy=multi-user.target

docs/installation/debian.rst

@@ -24,7 +24,7 @@ On Debian-like systems, you can install them using:
     # Install dependencies
     sudo apt-get install curl python3-pip python3-venv git unzip libldap2-dev libsasl2-dev
     # Funkwhale dependencies
-    sudo apt install build-essential ffmpeg libjpeg-dev libmagic-dev libpq-dev postgresql-client python3-dev
+    sudo apt install build-essential ffmpeg libjpeg-dev libmagic-dev libpq-dev postgresql-client python3-dev make
 
 On Arch Linux and its derivatives: