From 4df883678b3ffad9e38c6efeb8abe5721f50d596 Mon Sep 17 00:00:00 2001 From: Janek <6456-xeruf@users.noreply.dev.funkwhale.audio> Date: Thu, 25 Nov 2021 23:57:42 +0000 Subject: [PATCH] docs: wording overhaul, improve Caddy info, update music importing MR !1385 --- docs/admin/importing-music.rst | 146 +++++++++++++++++++-------------- docs/installation/index.rst | 70 +++++++++------- 2 files changed, 126 insertions(+), 90 deletions(-) diff --git a/docs/admin/importing-music.rst b/docs/admin/importing-music.rst index 5ba7a567d..27e303195 100644 --- a/docs/admin/importing-music.rst +++ b/docs/admin/importing-music.rst @@ -1,21 +1,21 @@ Importing music from the server =============================== -Funkwhale can import music files that are located on the server assuming -they readable by the Funkwhale application. Your music files should contain at -least an ``artist``, ``album`` and ``title`` tags, but we recommend you tag -it extensively using a proper tool, such as Beets or Musicbrainz Picard. +Funkwhale can import music files saved on the server +assuming they are readable by the Funkwhale application. +Your music files should contain at least +``artist``, ``album`` and ``title`` tags, +but we recommend you tag extensively using a proper tool, +such as Beets or Musicbrainz Picard. Funkwhale supports two different import modes: -- copy (the default): files are copied into Funkwhale's internal storage. This means importing a 1GB library will result in the same amount of space being used by Funkwhale. -- :ref:`in-place ` (when the ``--in-place`` is provided): files are referenced in Funkwhale's DB but not copied or touched in anyway. This is useful if you have a huge library, or one that is updated by an external tool such as Beets. +- copy(default): files are copied into Funkwhale's internal storage. This means importing a 1GB library will result in the same amount of space being used by Funkwhale. +- :ref:`in-place ` (with ``--in-place`` flag): files are referenced in Funkwhale's DB but not copied or touched in anyway. This is useful if you have a huge library, or one that is updated by an external tool such as Beets. -.. note:: - - In Funkwhale 1.0, **the default behaviour will change to in-place import** - -Regardless of the mode you're choosing, import works as described below, assuming your files are located in +Regardless of the mode you choose, +follow the below steps to import music, +assuming your files are located in ``/srv/funkwhale/data/music``: .. code-block:: bash @@ -24,17 +24,18 @@ Regardless of the mode you're choosing, import works as described below, assumin python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput .. note:: - You'll have to create a library in the Web UI before to get your library ID. Simply visit - https://yourdomain/content/libraries/ to create one. + You have to create a library in the Web UI to get your library ID. + Simply visit https://yourdomain/content/libraries/ to create one. - Library IDs are available in library urls or sharing link. In this example: + Library IDs are part of the library url or sharing link. + For example, the library ID of https://funkwhale.instance/content/libraries/769a2ae3-eb3d-4aff-9f94-2c4d80d5c2d1, - the library ID is 769a2bc3-eb1d-4aff-9f84-2c4d80d5c2d1 + is 769a2bc3-eb1d-4aff-9f84-2c4d80d5c2d1 You can use only the first characters of the ID when calling the command, like that: ``export LIBRARY_ID="769a2bc3"`` -When you use docker, the ``/srv/funkwhale/data/music`` is mounted from the host +When you use docker, ``/srv/funkwhale/data/music`` is mounted from the host to the ``/music`` directory on the container: .. code-block:: bash @@ -50,25 +51,24 @@ When you installed Funkwhale via ansible, you need to call a script instead of P /srv/funkwhale/manage import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput -The import command supports several options, and you can check the help to -get details:: +The import command supports several options, +check the help for details:: docker-compose run --rm api python manage.py import_files --help .. note:: - For the best results, we recommend tagging your music collection through - `Picard `_ in order to have the best quality metadata. + We recommend tagging your music collection using `Picard `_ to have the best quality metadata. .. note:: - This command is idempotent, meaning you can run it multiple times on the same - files and already imported files will simply be skipped. + This command is idempotent, + meaning you can run it multiple times on the same files + and already imported files are simply skipped. .. note:: - At the moment, only Flac, OGG/Vorbis and MP3 or AIFF files with ID3 tags are supported - + At the moment, only Flac, OGG/Vorbis and MP3 or AIFF files with ID3 tags are supported. .. _in-place-import: @@ -76,49 +76,60 @@ get details:: In-place import ^^^^^^^^^^^^^^^ -By default, the CLI-importer will copy imported files to Funkwhale's internal -storage. This means importing a 1GB library will result in the same amount -of space being used by Funkwhale. +By default, the CLI-importer will copy imported files to Funkwhale's internal storage. +This means importing a 1GB library will result +in the same amount of space being used by Funkwhale. While this behaviour has some benefits (easier backups and configuration), -it's not always the best choice, especially if you have a huge library -to import and don't want to double your disk usage. +it is not always the best choice, +especially if you have a huge library to import +and don't want to double your disk usage. -The CLI importer supports an additional ``--in-place`` option that triggers the -following behaviour during import: +The CLI importer supports an additional ``--in-place`` option +through which Funkwhale will store file paths rather than file content. -1. Imported files are not store in Funkwhale anymore -2. Instead, Funkwhale will store the file path and use it to serve the music +Structure +********* -Because those files are not managed by Funkwhale, we offer additional -configuration options to ensure the webserver can serve them properly: +Because imported files are not managed by Funkwhale, +we offer additional configuration options +to ensure the webserver can serve them properly: - :data:`MUSIC_DIRECTORY_PATH ` - :data:`MUSIC_DIRECTORY_SERVING_PATH ` We recommend you symlink all your music directories into ``/srv/funkwhale/data/music`` -and run the `import_files` command from that directory. This will make it possible -to use multiple music directories, without any additional configuration -on the webserver side. +and run the `import_files` command from that directory. +This will make it possible to use multiple music directories +without any additional configuration on the webserver side. -For instance, if you have a NFS share with your music mounted at ``/media/mynfsshare``, +For instance, if you have an NFS share +with your music mounted at ``/media/mynfsshare``, you can create a symlink like this:: ln -s /media/mynfsshare /srv/funkwhale/data/music/nfsshare -And import music from this share with this command:: +And import music from the share:: export LIBRARY_ID="" python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/nfsshare/" --recursive --noinput --in-place -On docker setups, it will require a bit more work, because while the ``/srv/funkwhale/data/music`` is mounted -in containers, symlinked directories are not. +Docker +****** -To fix that, you can use bind mounts instead of symbolic links, as it replicates the source directory tree. With the previous NFS share, it would go this way:: +Docker setups require a bit more work, +because while the ``/srv/funkwhale/data/music`` is mounted in containers, +symlinked directories are not. + +To fix that, you can use bind mounts instead of symbolic links, +as they replicate the source directory tree. +With the previous NFS share, use this command:: mount --bind /media/mynfsshare /srv/funkwhale/data/music/nfsshare -If you want to go with symlinks, ensure each symlinked directory is mounted as a volume as well in your ``docker-compose.yml`` file:: +If you want to go with symlinks, +ensure each symlinked directory is mounted as a volume +as well as in your ``docker-compose.yml`` file:: celeryworker: volumes: @@ -137,15 +148,16 @@ If you want to go with symlinks, ensure each symlinked directory is mounted as a Metadata updates ^^^^^^^^^^^^^^^^ -When doing an import with in ``in-place`` mode, the importer will also check and update existing entries -found in the database. For instance, if a file was imported, the ID3 Title tag was updated, and you rerun a scan, -Funkwhale will pick up the new title. The following fields can be updated this way: +When doing an import with in ``in-place`` mode, +the importer will also check and update existing entries found in the database. +For instance, if the ID3 Title tag of an existing song was updated since the last scan, Funkwhale picks up the new title. +The following fields can be updated this way: - Track mbid - Track title - Track position and disc number - Track license and copyright -- Track genre +- Track genre (`from version 1.2 `_) - Album cover - Album title - Album mbid @@ -155,17 +167,23 @@ Funkwhale will pick up the new title. The following fields can be updated this w - Album artist name - Album artist mbid +Changes in artist name can lead to multiple artists with the same name in the database, +`this is a known issue `_ +and can be remedied by adding mbids. React to filesystem events with ``--watch`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If you have a really big library or one that is updated quite often, running the ``import_files`` command by hand -may not be practical. To help with this use case, the ``import_files`` command supports a ``--watch`` flag that will observes filesystem events -instead of performing a full import. +If you have a really big library or update it regularly, +running the ``import_files`` command by hand may not be practical. +For this use case, +the ``import_files`` command supports a ``--watch`` flag +through which it observes filesystem events instead of performing a full import. -File creation, move, update and removal are handled when ``--watch`` is provided: +File creation, move, update and removal +are handled when ``--watch`` is provided: -- Files created in the watched directory are imported immediatly +- Files created in the watched directory are imported immediately - If using ``in-place`` mode, files updates trigger a metadata update on the corresponding entries - If using ``in-place`` mode, files that are moved and known by Funkwhale will see their path updated in Funkwhale's DB - If using ``in-place`` mode, files that are removed and known by Funkwhale will be removed from Funkwhale's DB @@ -173,7 +191,8 @@ File creation, move, update and removal are handled when ``--watch`` is provided Pruning dangling metadata with ``--prune`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Funkwhale is, by design, conservative with music metadata in its database. If you remove a file from Funkwhale's DB, +Funkwhale is, by design, conservative with music metadata in its database. +If you remove a file from Funkwhale's DB, the corresponding artist, album and track object won't be deleted by default. If you want to prune dangling metadata from the database once the ``import_files`` command is over, simply add the ``--prune`` flag. @@ -182,18 +201,20 @@ This also works in with ``--watch``. Album covers ^^^^^^^^^^^^ -Whenever possible, Funkwhale will import album cover, with the following precedence: +Whenever possible, Funkwhale obtains album covers for tracks, +with the following precedence: -1. It will use the cover embedded in the audio files themeselves, if any (Flac/MP3 only) -2. It will use a cover.jpg or a cover.png file from the imported track directory, if any -3. It will fetch cover art from musicbrainz, assuming the file is tagged correctly +1. The cover embedded in the audio files themeselves, if any (Flac/MP3 only) +2. Use a cover.jpg or a cover.png file from the imported track directory, if any +3. Fetch cover art from musicbrainz, assuming the file is tagged correctly Getting demo tracks ^^^^^^^^^^^^^^^^^^^ -If you do not have any music on your server but still want to test the import -process, you can call the following methods do download a few albums licenced -under creative commons (courtesy of Jamendo): +If you do not have any music on your server +but want to test the import process, +you can call the following methods +to download a few albums licenced under creative commons (courtesy of Jamendo): .. parsed-literal:: @@ -202,4 +223,5 @@ under creative commons (courtesy of Jamendo): chmod +x download-tracks.sh ./download-tracks.sh music.txt -This will download a bunch of zip archives (one per album) under the ``data/music`` directory and unzip their content. +This will download a bunch of zip archives (one per album) +under the ``data/music`` directory and unzip their content. diff --git a/docs/installation/index.rst b/docs/installation/index.rst index ce9a77031..b951d6227 100644 --- a/docs/installation/index.rst +++ b/docs/installation/index.rst @@ -4,15 +4,17 @@ Installation Requirements ------------ -Regardless of your chosen installation method, the following requirements must be met in order to successfully deploy Funkwhale: +Regardless of your chosen installation method, +the following is required to successfully deploy Funkwhale: - **A dedicated domain or subdomain**: it is not possible to deploy Funkwhale on a subdirectory of an existing domain. - **Access to ports 80 and/or 443**: if you cannot serve the Funkwhale web app and API on these ports, federation will not work .. note:: - Because of the federated nature of Funkwhale, **it is strongly recommended not to change the Funkwhale domain after initial deployment**, as it is likely to break - your installation. + Because of the federated nature of Funkwhale, + **it is strongly recommended not to change the Funkwhale domain after initial deployment**, + as it is likely to break your installation. Project architecture -------------------- @@ -34,8 +36,9 @@ The project relies on the following components and services to work: Hardware requirements --------------------- -Funkwhale is not especially CPU hungry. On a dockerized instance with 2 CPUs -and a few active users, the memory footprint is around 500Mb:: +Funkwhale is not especially CPU hungry. +On a dockerized instance with 2 CPUs and a few active users, +the memory footprint is around 500Mb:: CONTAINER MEM USAGE funkwhale_api_1 202 MiB @@ -148,10 +151,11 @@ Serving only the frontend .. note:: - You do not need to do this if you are deploying using Docker, as frontend files - are already included in the docker image. + You do not need to do this if you are deploying using Docker, + as frontend files are already included in the docker image. - You also do not need to do this if you are deploying manually on Debian or Arch, as this is covered by the corresponding documentation already. + You also do not need to do this if you are deploying manually on Debian or Arch, + as this is covered by the corresponding documentation already. Files for the web frontend are purely static and can simply be downloaded, unzipped and served from any webserver: @@ -167,20 +171,23 @@ Files for the web frontend are purely static and can simply be downloaded, unzip Reverse proxy configuration --------------------------- -In order to make Funkwhale accessible from outside your server and to play nicely with other applications on your machine, you should configure a reverse proxy. +In order to make Funkwhale accessible from outside your server +and to play nicely with other applications on your machine, +you should configure a reverse proxy. We offer sample configurations for Nginx, Apache2 and Caddy. .. note:: - You can freely adapt the proposed configuration to your own needs, as we cannot - cover every use case with a single template, especially when it's related - to SSL configuration. + You can freely adapt the proposed configuration to your own needs, + as we cannot cover every use case with a single template, + especially when it's related to SSL configuration. Nginx ^^^^^ -Ensure you have a recent version of nginx on your server. On Debian-like system, you would have to run the following: +Ensure you have a recent version of nginx on your server. +On a Debian-based system use apt: .. code-block:: shell @@ -193,9 +200,11 @@ On Arch Linux and its derivatives: sudo pacman -S nginx -To avoid configuration errors at this level, we will generate an nginx configuration -using your .env file. This will ensure your reverse-proxy configuration always -match the application configuration and make upgrade/maintenance easier. +To avoid configuration errors at this level, +we will generate an nginx configuration using your .env file. +This will ensure your reverse-proxy configuration +always matches the application configuration +and makes upgrade/maintenance easier. .. note:: The following commands need to be run as superuser. @@ -297,7 +306,8 @@ Add the following to your ``.env`` file:: REVERSE_PROXY_TYPE=apache2 -Then restart Funkwhale. This is needed to ensure Funkwhale provides proper headers for media file serving. +Then restart Funkwhale. +This is needed to ensure Funkwhale provides proper headers for media file serving. Then, download our sample virtualhost file: @@ -306,18 +316,23 @@ Then, download our sample virtualhost file: curl -L -o /etc/apache2/sites-available/funkwhale.conf "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/|version|/deploy/apache.conf" ln -s /etc/apache2/sites-available/funkwhale.conf /etc/apache2/sites-enabled/ -You can tweak the configuration file according to your setup, especially the -TLS configuration. Otherwise, defaults should work if you followed the -installation guide. +Tweak the configuration file according to your setup, +especially the TLS configuration. +Otherwise, defaults should work if you followed the installation guide. -Check the configuration is valid with ``apache2ctl configtest``, and once you're -done, load the new configuration with ``service apache2 restart``. +Check the configuration with ``apache2ctl configtest`` +and once you're done, +load the new configuration with ``service apache2 restart``. Caddy ^^^^^ -If you're using Caddy as a reverse proxy in front of your docker containers (either mono or multi-container setup), -you can use the following Caddyfile configuration: +We currently do not support a Caddy-only setup, but you can +`help develop it `_! + +To employ Caddy as a reverse proxy in front of your docker containers +(either mono- or multi-container setup), +use the following Caddyfile configuration: Caddy v2:: @@ -338,7 +353,9 @@ Caddy v1:: HTTPS configuration ^^^^^^^^^^^^^^^^^^^ -After configuring the reverse proxy, you need a SSL certificate to enable HTTPS on your server. +After configuring the reverse proxy, +you need a SSL certificate to enable HTTPS on your server +(unless you use Caddy, which handles them automatically). The default reverse proxy configuration assumes you have those available at ``/etc/letsencrypt/live/${FUNKWHALE_HOSTNAME}/``, which is the path used by `certbot `_ when generating certificates with Let's Encrypt. @@ -365,6 +382,3 @@ a certificate, as shown below. These instructions are provided by `certbot