diff --git a/.env.dev b/.env.dev index c09262509..d9e2dd3ce 100644 --- a/.env.dev +++ b/.env.dev @@ -1,4 +1,3 @@ -API_AUTHENTICATION_REQUIRED=True RAVEN_ENABLED=false RAVEN_DSN=https://44332e9fdd3d42879c7d35bf8562c6a4:0062dc16a22b41679cd5765e5342f716@sentry.eliotberriot.com/5 DJANGO_ALLOWED_HOSTS=.funkwhale.test,localhost,nginx,0.0.0.0,127.0.0.1 diff --git a/.gitignore b/.gitignore index 548cfd7b3..25b088739 100644 --- a/.gitignore +++ b/.gitignore @@ -89,3 +89,4 @@ data/ .env po/*.po +docs/swagger diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6a0f4b9d8..5f65e60da 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -92,12 +92,14 @@ build_front: pages: stage: test - image: python:3.6-alpine + image: python:3.6 + variables: + BUILD_PATH: "../public" before_script: - cd docs script: - pip install sphinx - - python -m sphinx . ../public + - ./build_docs.sh artifacts: paths: - public diff --git a/CHANGELOG b/CHANGELOG index c56d58836..82c867bf8 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,8 +1,120 @@ Changelog ========= +You can subscribe to release announcements by: + +- Following `funkwhale@mastodon.eliotberriot.com `_ on Mastodon +- Subscribing to the following Atom feed: https://code.eliotberriot.com/funkwhale/funkwhale/commits/develop?format=atom&search=tag + +This changelog is viewable on the web at https://docs.funkwhale.audio/changelog.html. + .. towncrier +0.11 (unreleased) +----------------- + +Upgrade instructions are available at https://docs.funkwhale.audio/upgrading.html + +Special thanks for this release go to @renon:matrix.org (@Hazmo on Gitlab) +for bringing Apache2 support to Funkwhale and contributing on other issues. +Thank you! + +Features: + +- Funkwhale now works behind an Apache2 reverse proxy (!165) + check out the brand new documentation at https://docs.funkwhale.audio/installation/index.html#apache2 + if you want to try it! +- Users can now request password reset by email, assuming a SMTP server was + correctly configured (#187) + +Enhancements: + +- Added a fix_track_files command to run checks and fixes against library + (#183) +- Avoid fetching Actor object on every request authentication +- Can now relaunch errored jobs and batches (#176) +- List pending requests by default, added a status filter for requests (#109) +- More structured menus in sidebar, added labels with notifications +- Sample virtual-host file for Apache2 reverse-proxy (!165) +- Store high-level settings (such as federation or auth-related ones) in + database (#186) + + +Bugfixes: + +- Ensure in place imported files get a proper mimetype (#183) +- Federation cache suppression is now simpler and also deletes orphaned files + (#189) +- Fixed small UI glitches/bugs in federation tabs (#184) +- X-sendfile not working with in place import (#182) + + +Documentation: + +- Added a documentation area for third-party projects (#180) +- Added documentation for optimizing Funkwhale and reduce its memory footprint. +- Document that the database should use an utf-8 encoding (#185) +- Foundations for API documentation with Swagger (#178) + + +Database storage for high-level settings +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Due to the work done in #186, the following environment variables have been +deprecated: + +- FEDERATION_ENABLED +- FEDERATION_COLLECTION_PAGE_SIZE +- FEDERATION_MUSIC_NEEDS_APPROVAL +- FEDERATION_ACTOR_FETCH_DELAY +- PLAYLISTS_MAX_TRACKS +- API_AUTHENTICATION_REQUIRED + +Configuration for this settings has been moved to database, as it will provide +a better user-experience, by allowing you to edit these values on-the-fly, +without restarting Funkwhale processes. + +You can leave those environment variables in your .env file for now, as the +values will be used to populate the database entries. We'll make a proper +announcement when the variables won't be used anymore. + +Please browse https://docs.funkwhale.audio/configuration.html#instance-settings +for more information about instance configuration using the web interface. + + +System emails +^^^^^^^^^^^^^ + +Starting from this release, Funkwhale will send two types +of emails: + +- Email confirmation emails, to ensure a user's email is valid +- Password reset emails, enabling user to reset their password without an admin's intervention + +Email sending is disabled by default, as it requires additional configuration. +In this mode, emails are simply outputed on stdout. + +If you want to actually send those emails to your users, you should edit your +.env file and tweak the EMAIL_CONFIG variable. See :ref:`setting-EMAIL_CONFIG` +for more details. + +.. note:: + + As a result of these changes, the DJANGO_EMAIL_BACKEND variable, + which was not documented, has no effect anymore. You can safely remove it from + your .env file if it is set. + + +Proxy headers for non-docker deployments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For non-docker deployments, add ``--proxy-headers`` at the end of the ``daphne`` +command in :file:`/etc/systemd/system/funkwhale-server.service`. + +This will ensure the application receive the correct IP address from the client +and not the proxy's one. + + 0.10 (2018-04-23) ----------------- diff --git a/CONTRIBUTING b/CONTRIBUTING new file mode 100644 index 000000000..9f4ec8850 --- /dev/null +++ b/CONTRIBUTING @@ -0,0 +1,442 @@ +Contibute to Funkwhale development +================================== + +First of all, thank you for your interest in the project! We really +appreciate the fact that you're about to take some time to read this +and hack on the project. + +This document will guide you through common operations such as: + +- Setup your development environment +- Working on your first issue +- Writing unit tests to validate your work +- Submit your work + + +Setup your development environment +---------------------------------- + +If you want to fix a bug or implement a feature, you'll need +to run a local, development copy of funkwhale. + +We provide a docker based development environment, which should +be both easy to setup and work similarly regardless of your +development machine setup. + +Instructions for bare-metal setup will come in the future (Merge requests +are welcome). + +Installing docker and docker-compose +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This is already cover in the relevant documentations: + +- https://docs.docker.com/install/ +- https://docs.docker.com/compose/install/ + +Cloning the project +^^^^^^^^^^^^^^^^^^^ + +Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH:: + + git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git + cd funkwhale + + +A note about branches +^^^^^^^^^^^^^^^^^^^^^ + +Next release development occurs on the "develop" branch, and releases are made on the "master" branch. Therefor, when submitting Merge Requests, ensure you are merging on the develop branch. + + +Working with docker +^^^^^^^^^^^^^^^^^^^ + +In developpement, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this:: + + docker-compose -f dev.yml logs + +If you do not want to add the ``-f dev.yml`` snippet everytime, you can run this command before starting your work:: + + export COMPOSE_FILE=dev.yml + + +Building the containers +^^^^^^^^^^^^^^^^^^^^^^^ + +On your initial clone, or if there have been some changes in the +app dependencies, you will have to rebuild your containers. This is done +via the following command:: + + docker-compose -f dev.yml build + + +Creating your env file +^^^^^^^^^^^^^^^^^^^^^^ + +We provide a working .env.dev configuration file that is suitable for +development. However, to enable customization on your machine, you should +also create a .env file that will hold your personal environment +variables (those will not be commited to the project). + +Create it like this:: + + touch .env + + +Database management +^^^^^^^^^^^^^^^^^^^ + +To setup funkwhale's database schema, run this:: + + docker-compose -f dev.yml run --rm api python manage.py migrate + +This will create all the tables needed for the API to run proprely. +You will also need to run this whenever changes are made on the database +schema. + +It is safe to run this command multiple times, so you can run it whenever +you fetch develop. + + +Development data +^^^^^^^^^^^^^^^^ + +You'll need at least an admin user and some artists/tracks/albums to work +locally. + +Create an admin user with the following command:: + + docker-compose -f dev.yml run --rm api python manage.py createsuperuser + +Injecting fake data is done by running the fllowing script:: + + artists=25 + command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)" + echo $command | docker-compose -f dev.yml run --rm api python manage.py shell -i python + +The previous command will create 25 artists with random albums, tracks +and metadata. + + +Launch all services +^^^^^^^^^^^^^^^^^^^ + +Then you can run everything with:: + + docker-compose -f dev.yml up + +This will launch all services, and output the logs in your current terminal window. +If you prefer to launch them in the background instead, use the ``-d`` flag, and access the logs when you need it via ``docker-compose -f dev.yml logs --tail=50 --follow``. + +Once everything is up, you can access the various funkwhale's components: + +- The Vue webapp, on http://localhost:8080 +- The API, on http://localhost:8080/api/v1/ +- The django admin, on http://localhost:8080/api/admin/ + +Stopping everything +^^^^^^^^^^^^^^^^^^^ + +Once you're down with your work, you can stop running containers, if any, with:: + + docker-compose -f dev.yml stop + + +Removing everything +^^^^^^^^^^^^^^^^^^^ + +If you want to wipe your development environment completely (e.g. if you want to start over from scratch), just run:: + + docker-compose -f dev.yml down -v + +This will wipe your containers and data, so please be careful before running it. + +You can keep your data by removing the ``-v`` flag. + + +Working with federation locally +------------------------------- + +This is not needed unless you need to work on federation-related features. + +To achieve that, you'll need: + +1. to update your dns resolver to resolve all your .dev hostnames locally +2. a reverse proxy (such as traefik) to catch those .dev requests and + and with https certificate +3. two instances (or more) running locally, following the regular dev setup + +Resolve .dev names locally +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you use dnsmasq, this is as simple as doing:: + + echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf + sudo systemctl restart dnsmasq + +If you use NetworkManager with dnsmasq integration, use this instead:: + + echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf + sudo systemctl restart NetworkManager + +Add wildcard certificate to the trusted certificates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Simply copy bundled certificates:: + + sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/ + sudo update-ca-certificates + +This certificate is a wildcard for ``*.funkwhale.test`` + +Run a reverse proxy for your instances +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Create docker network +^^^^^^^^^^^^^^^^^^^^ + +Create the federation network:: + + docker network create federation + +Launch everything +^^^^^^^^^^^^^^^^^ + +Launch the traefik proxy:: + + docker-compose -f docker/traefik.yml up -d + +Then, in separate terminals, you can setup as many different instances as you +need:: + + export COMPOSE_PROJECT_NAME=node2 + docker-compose -f dev.yml run --rm api python manage.py migrate + docker-compose -f dev.yml run --rm api python manage.py createsuperuser + docker-compose -f dev.yml up nginx api front nginx api celeryworker + +Note that by default, if you don't export the COMPOSE_PROJECT_NAME, +we will default to node1 as the name of your instance. + +Assuming your project name is ``node1``, your server will be reachable +at ``https://node1.funkwhale.test/``. Not that you'll have to trust +the SSL Certificate as it's self signed. + +When working on federation with traefik, ensure you have this in your ``env``:: + + # This will ensure we don't bind any port on the host, and thus enable + # multiple instances of funkwhale to be spawned concurrently. + WEBPACK_DEVSERVER_PORT_BINDING= + # This disable certificate verification + EXTERNAL_REQUESTS_VERIFY_SSL=false + # this ensure you don't have incorrect urls pointing to http resources + FUNKWHALE_PROTOCOL=https + + +Typical workflow for a contribution +----------------------------------- + +0. Fork the project if you did not already or if you do not have access to the main repository +1. Checkout the development branch and pull most recent changes: ``git checkout develop && git pull`` +2. If working on an issue, assign yourself to the issue. Otherwise, consider open an issue before starting to work on something, especially for new features. +3. Create a dedicated branch for your work ``42-awesome-fix``. It is good practice to prefix your branch name with the ID of the issue you are solving. +4. Work on your stuff +5. Commit small, atomic changes to make it easier to review your contribution +6. Add a changelog fragment to summarize your changes: ``echo "Implemented awesome stuff (#42)" > changes/changelog.d/42.feature"`` +7. Push your branch +8. Create your merge request +9. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute! + + +Internationalization +-------------------- + +When working on the front-end, any end-user string should be translated +using either ```` or the ``$t('yourstring')`` +function. + +Extraction is done by calling ``yarn run i18n-extract``, which +will pull all the strings from source files and put them in a PO file. + +Contributing to the API +----------------------- + +Project structure +^^^^^^^^^^^^^^^^^ + +.. code-block:: shell + + tree api -L 2 -d + api + ├── config # configuration directory (settings, urls, wsgi server) + │ └── settings # Django settings files + ├── funkwhale_api # project directory, all funkwhale logic is here + ├── requirements # python requirements files + └── tests # test files, matches the structure of the funkwhale_api directory + +.. note:: + + Unless trivial, API contributions must include unittests to ensure + your fix or feature is working as expected and won't break in the future + +Running tests +^^^^^^^^^^^^^ + +To run the pytest test suite, use the following command:: + + docker-compose -f dev.yml run --rm api pytest + +This is regular pytest, so you can use any arguments/options that pytest usually accept:: + + # get some help + docker-compose -f dev.yml run --rm api pytest -h + # Stop on first failure + docker-compose -f dev.yml run --rm api pytest -x + # Run a specific test file + docker-compose -f dev.yml run --rm api pytest tests/test_acoustid.py + +Writing tests +^^^^^^^^^^^^^ + +Although teaching you how to write unit tests is outside of the scope of this +document, you'll find below a collection of tips, snippets and resources +you can use if you want to learn on that subject. + +Useful links: + +- `A quick introduction to unit test writing with pytest `_ +- `A complete guide to Test-Driven Development (although not using Pytest) `_ +- `pytest `_: documentation of our testing engine and runner +- `pytest-mock `_: project page of our mocking engine +- `factory-boy `_: documentation of factory-boy, which we use to easily generate fake objects and data + +Recommendations: + +- Test files must target a module and mimic ``funkwhale_api`` directory structure: if you're writing tests for ``funkwhale_api/myapp/views.py``, you should put thoses tests in ``tests/myapp/test_views.py`` +- Tests should be small and test one thing. If you need to test multiple things, write multiple tests. + +We provide a lot of utils and fixtures to make the process of writing tests as +painless as possible. You'll find some usage examples below. + +Use factories to create arbitrary objects: + +.. code-block:: python + + # funkwhale_api/myapp/users.py + + def downgrade_user(user): + """ + A simple function that remove superuser status from users + and return True if user was actually downgraded + """ + downgraded = user.is_superuser + user.is_superuser = False + user.save() + return downgraded + + # tests/myapp/test_users.py + from funkwhale_api.myapp import users + + def test_downgrade_superuser(factories): + user = factories['users.User'](is_superuser=True) + downgraded = users.downgrade_user(user) + + assert downgraded is True + assert user.is_superuser is False + + def test_downgrade_normal_user_does_nothing(factories): + user = factories['users.User'](is_superuser=False) + downgraded = something.downgrade_user(user) + + assert downgraded is False + assert user.is_superuser is False + +.. note:: + + We offer factories for almost if not all models. Factories are located + in a ``factories.py`` file inside each app. + +Mocking: mocking is the process of faking some logic in our code. This is +useful when testing components that depend on each other: + +.. code-block:: python + + # funkwhale_api/myapp/notifications.py + + def notify(email, message): + """ + A function that sends an email to the given recipient + with the given message + """ + + # our email sending logic here + # ... + + # funkwhale_api/myapp/users.py + from . import notifications + + def downgrade_user(user): + """ + A simple function that remove superuser status from users + and return True if user was actually downgraded + """ + downgraded = user.is_superuser + user.is_superuser = False + user.save() + if downgraded: + notifications.notify(user.email, 'You have been downgraded!') + return downgraded + + # tests/myapp/test_users.py + def test_downgrade_superuser_sends_email(factories, mocker): + """ + Your downgrade logic is already tested, however, we want to ensure + an email is sent when user is downgraded, but we don't have any email + server available in our testing environment. Thus, we need to mock + the email sending process. + """ + mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify') + user = factories['users.User'](is_superuser=True) + users.downgrade_user(user) + + # here, we ensure our notify function was called with proper arguments + mocked_notify.assert_called_once_with(user.email, 'You have been downgraded') + + + def test_downgrade_not_superuser_skips_email(factories, mocker): + mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify') + user = factories['users.User'](is_superuser=True) + users.downgrade_user(user) + + # here, we ensure no email was sent + mocked_notify.assert_not_called() + +Views: you can find some readable views tests in :file:`tests/users/test_views.py` + +.. note:: + + A complete list of available-fixtures is available by running + ``docker-compose -f dev.yml run --rm api pytest --fixtures`` + + +Contributing to the front-end +----------------------------- + +Running tests +^^^^^^^^^^^^^ + +To run the front-end test suite, use the following command:: + + docker-compose -f dev.yml run --rm front yarn run unit + +We also support a "watch and test" mode were we continually relaunch +tests when changes are recorded on the file system:: + + docker-compose -f dev.yml run --rm front yarn run unit-watch + +The latter is especially useful when you are debugging failing tests. + +.. note:: + + The front-end test suite coverage is still pretty low diff --git a/README.rst b/README.rst index 8a0ea4932..8646527ad 100644 --- a/README.rst +++ b/README.rst @@ -1,6 +1,10 @@ Funkwhale ============= +.. image:: ./front/src/assets/logo/logo-full-500.png + :alt: Funkwhale logo + :target: https://funkwhale.audio + A self-hosted tribute to Grooveshark.com. LICENSE: BSD @@ -8,289 +12,18 @@ LICENSE: BSD Getting help ------------ -We offer various Matrix.org rooms to discuss about funkwhale: +We offer various Matrix.org rooms to discuss about Funkwhale: - `#funkwhale:matrix.org `_ for general questions about funkwhale - `#funkwhale-dev:matrix.org `_ for development-focused discussion Please join those rooms if you have any questions! -Running the development version -------------------------------- +You can also contact `@funkwhale@mastodon.eliotberriot.com `_ on the fediverse. -If you want to fix a bug or implement a feature, you'll need -to run a local, development copy of funkwhale. -We provide a docker based development environment, which should -be both easy to setup and work similarly regardless of your -development machine setup. +Contribute +---------- -Instructions for bare-metal setup will come in the future (Merge requests -are welcome). - -Installing docker and docker-compose -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This is already cover in the relevant documentations: - -- https://docs.docker.com/install/ -- https://docs.docker.com/compose/install/ - -Cloning the project -^^^^^^^^^^^^^^^^^^^ - -Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH:: - - git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git - cd funkwhale - - -A note about branches -^^^^^^^^^^^^^^^^^^^^^ - -Next release development occurs on the "develop" branch, and releases are made on the "master" branch. Therefor, when submitting Merge Requests, ensure you are merging on the develop branch. - - -Working with docker -^^^^^^^^^^^^^^^^^^^ - -In developpement, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this:: - - docker-compose -f dev.yml logs - -If you do not want to add the ``-f dev.yml`` snippet everytime, you can run this command before starting your work:: - - export COMPOSE_FILE=dev.yml - - -Building the containers -^^^^^^^^^^^^^^^^^^^^^^^ - -On your initial clone, or if there have been some changes in the -app dependencies, you will have to rebuild your containers. This is done -via the following command:: - - docker-compose -f dev.yml build - - -Creating your env file -^^^^^^^^^^^^^^^^^^^^^^ - -We provide a working .env.dev configuration file that is suitable for -development. However, to enable customization on your machine, you should -also create a .env file that will hold your personal environment -variables (those will not be commited to the project). - -Create it like this:: - - touch .env - - -Database management -^^^^^^^^^^^^^^^^^^^ - -To setup funkwhale's database schema, run this:: - - docker-compose -f dev.yml run --rm api python manage.py migrate - -This will create all the tables needed for the API to run proprely. -You will also need to run this whenever changes are made on the database -schema. - -It is safe to run this command multiple times, so you can run it whenever -you fetch develop. - - -Development data -^^^^^^^^^^^^^^^^ - -You'll need at least an admin user and some artists/tracks/albums to work -locally. - -Create an admin user with the following command:: - - docker-compose -f dev.yml run --rm api python manage.py createsuperuser - -Injecting fake data is done by running the fllowing script:: - - artists=25 - command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)" - echo $command | docker-compose -f dev.yml run --rm api python manage.py shell -i python - -The previous command will create 25 artists with random albums, tracks -and metadata. - - -Launch all services -^^^^^^^^^^^^^^^^^^^ - -Then you can run everything with:: - - docker-compose -f dev.yml up - -This will launch all services, and output the logs in your current terminal window. -If you prefer to launch them in the background instead, use the ``-d`` flag, and access the logs when you need it via ``docker-compose -f dev.yml logs --tail=50 --follow``. - -Once everything is up, you can access the various funkwhale's components: - -- The Vue webapp, on http://localhost:8080 -- The API, on http://localhost:8080/api/v1/ -- The django admin, on http://localhost:8080/api/admin/ - - -Running API tests -^^^^^^^^^^^^^^^^^ - -To run the pytest test suite, use the following command:: - - docker-compose -f dev.yml run --rm api pytest - -This is regular pytest, so you can use any arguments/options that pytest usually accept:: - - # get some help - docker-compose -f dev.yml run --rm api pytest -h - # Stop on first failure - docker-compose -f dev.yml run --rm api pytest -x - # Run a specific test file - docker-compose -f dev.yml run --rm api pytest tests/test_acoustid.py - - -Running front-end tests -^^^^^^^^^^^^^^^^^^^^^^^ - -To run the front-end test suite, use the following command:: - - docker-compose -f dev.yml run --rm front yarn run unit - -We also support a "watch and test" mode were we continually relaunch -tests when changes are recorded on the file system:: - - docker-compose -f dev.yml run --rm front yarn run unit-watch - -The latter is especially useful when you are debugging failing tests. - -.. note:: - - The front-end test suite coverage is still pretty low - - -Stopping everything -^^^^^^^^^^^^^^^^^^^ - -Once you're down with your work, you can stop running containers, if any, with:: - - docker-compose -f dev.yml stop - - -Removing everything -^^^^^^^^^^^^^^^^^^^ - -If you want to wipe your development environment completely (e.g. if you want to start over from scratch), just run:: - - docker-compose -f dev.yml down -v - -This will wipe your containers and data, so please be careful before running it. - -You can keep your data by removing the ``-v`` flag. - - -Typical workflow for a merge request -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -0. Fork the project if you did not already or if you do not have access to the main repository -1. Checkout the development branch and pull most recent changes: ``git checkout develop && git pull`` -2. Create a dedicated branch for your work ``42-awesome-fix``. It is good practice to prefix your branch name with the ID of the issue you are solving. -3. Work on your stuff -4. Commit small, atomic changes to make it easier to review your contribution -5. Add a changelog fragment to summarize your changes: ``echo "Implemented awesome stuff (#42)" > changes/changelog.d/42.feature"`` -6. Push your branch -7. Create your merge request -8. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute! - - -Internationalization --------------------- - -When working on the front-end, any end-user string should be translated -using either ```` or the ``$t('yourstring')`` -function. - -Extraction is done by calling ``yarn run i18n-extract``, which -will pull all the strings from source files and put them in a PO file. - - -Working with federation locally -------------------------------- - -To achieve that, you'll need: - -1. to update your dns resolver to resolve all your .dev hostnames locally -2. a reverse proxy (such as traefik) to catch those .dev requests and - and with https certificate -3. two instances (or more) running locally, following the regular dev setup - -Resolve .dev names locally -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you use dnsmasq, this is as simple as doing:: - - echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf - sudo systemctl restart dnsmasq - -If you use NetworkManager with dnsmasq integration, use this instead:: - - echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf - sudo systemctl restart NetworkManager - -Add wildcard certificate to the trusted certificates -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Simply copy bundled certificates:: - - sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/ - sudo update-ca-certificates - -This certificate is a wildcard for ``*.funkwhale.test`` - -Run a reverse proxy for your instances -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - -Create docker network -^^^^^^^^^^^^^^^^^^^^ - -Create the federation network:: - - docker network create federation - -Launch everything -^^^^^^^^^^^^^^^^^ - -Launch the traefik proxy:: - - docker-compose -f docker/traefik.yml up -d - -Then, in separate terminals, you can setup as many different instances as you -need:: - - export COMPOSE_PROJECT_NAME=node2 - docker-compose -f dev.yml run --rm api python manage.py migrate - docker-compose -f dev.yml run --rm api python manage.py createsuperuser - docker-compose -f dev.yml up nginx api front nginx api celeryworker - -Note that by default, if you don't export the COMPOSE_PROJECT_NAME, -we will default to node1 as the name of your instance. - -Assuming your project name is ``node1``, your server will be reachable -at ``https://node1.funkwhale.test/``. Not that you'll have to trust -the SSL Certificate as it's self signed. - -When working on federation with traefik, ensure you have this in your ``env``:: - - # This will ensure we don't bind any port on the host, and thus enable - # multiple instances of funkwhale to be spawned concurrently. - WEBPACK_DEVSERVER_PORT_BINDING= - # This disable certificate verification - EXTERNAL_REQUESTS_VERIFY_SSL=false - # this ensure you don't have incorrect urls pointing to http resources - FUNKWHALE_PROTOCOL=https +Contribution guidelines as well as development installation instructions +are outlined in `CONTRIBUTING `_ diff --git a/api/compose/django/daphne.sh b/api/compose/django/daphne.sh index 4fa304143..3ceb19e96 100755 --- a/api/compose/django/daphne.sh +++ b/api/compose/django/daphne.sh @@ -1,3 +1,3 @@ #!/bin/bash -eux python /app/manage.py collectstatic --noinput -/usr/local/bin/daphne -b 0.0.0.0 -p 5000 config.asgi:application +/usr/local/bin/daphne -b 0.0.0.0 -p 5000 config.asgi:application --proxy-headers diff --git a/api/config/settings/common.py b/api/config/settings/common.py index de1d653cb..f88aa5dd5 100644 --- a/api/config/settings/common.py +++ b/api/config/settings/common.py @@ -48,14 +48,20 @@ else: FUNKWHALE_URL = '{}://{}'.format(FUNKWHALE_PROTOCOL, FUNKWHALE_HOSTNAME) +# XXX: deprecated, see #186 FEDERATION_ENABLED = env.bool('FEDERATION_ENABLED', default=True) FEDERATION_HOSTNAME = env('FEDERATION_HOSTNAME', default=FUNKWHALE_HOSTNAME) +# XXX: deprecated, see #186 FEDERATION_COLLECTION_PAGE_SIZE = env.int( 'FEDERATION_COLLECTION_PAGE_SIZE', default=50 ) +# XXX: deprecated, see #186 FEDERATION_MUSIC_NEEDS_APPROVAL = env.bool( 'FEDERATION_MUSIC_NEEDS_APPROVAL', default=True ) +# XXX: deprecated, see #186 +FEDERATION_ACTOR_FETCH_DELAY = env.int( + 'FEDERATION_ACTOR_FETCH_DELAY', default=60 * 12) ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS') # APP CONFIGURATION @@ -138,7 +144,6 @@ INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS MIDDLEWARE = ( # Make sure djangosecure.middleware.SecurityMiddleware is listed first 'django.contrib.sessions.middleware.SessionMiddleware', - 'funkwhale_api.users.middleware.AnonymousSessionMiddleware', 'corsheaders.middleware.CorsMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', @@ -167,7 +172,22 @@ FIXTURE_DIRS = ( # EMAIL CONFIGURATION # ------------------------------------------------------------------------------ -EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.smtp.EmailBackend') + +# EMAIL +# ------------------------------------------------------------------------------ +DEFAULT_FROM_EMAIL = env( + 'DEFAULT_FROM_EMAIL', + default='Funkwhale '.format(FUNKWHALE_HOSTNAME)) + +EMAIL_SUBJECT_PREFIX = env( + "EMAIL_SUBJECT_PREFIX", default='[Funkwhale] ') +SERVER_EMAIL = env('SERVER_EMAIL', default=DEFAULT_FROM_EMAIL) + + +EMAIL_CONFIG = env.email_url( + 'EMAIL_CONFIG', default='consolemail://') + +vars().update(EMAIL_CONFIG) # DATABASE CONFIGURATION # ------------------------------------------------------------------------------ @@ -287,7 +307,7 @@ AUTHENTICATION_BACKENDS = ( 'django.contrib.auth.backends.ModelBackend', 'allauth.account.auth_backends.AuthenticationBackend', ) - +SESSION_COOKIE_HTTPONLY = False # Some really nice defaults ACCOUNT_AUTHENTICATION_METHOD = 'username_email' ACCOUNT_EMAIL_REQUIRED = True @@ -362,7 +382,7 @@ CORS_ORIGIN_ALLOW_ALL = True # 'funkwhale.localhost', # ) CORS_ALLOW_CREDENTIALS = True -API_AUTHENTICATION_REQUIRED = env.bool("API_AUTHENTICATION_REQUIRED", True) + REST_FRAMEWORK = { 'DEFAULT_PERMISSION_CLASSES': ( 'rest_framework.permissions.IsAuthenticated', @@ -377,6 +397,7 @@ REST_FRAMEWORK = { ), 'DEFAULT_AUTHENTICATION_CLASSES': ( 'funkwhale_api.common.authentication.JSONWebTokenAuthenticationQS', + 'funkwhale_api.common.authentication.BearerTokenHeaderAuth', 'rest_framework_jwt.authentication.JSONWebTokenAuthentication', 'rest_framework.authentication.SessionAuthentication', 'rest_framework.authentication.BasicAuthentication', @@ -386,6 +407,11 @@ REST_FRAMEWORK = { 'django_filters.rest_framework.DjangoFilterBackend', ) } +REST_AUTH_SERIALIZERS = { + 'PASSWORD_RESET_SERIALIZER': 'funkwhale_api.users.serializers.PasswordResetSerializer' # noqa +} +REST_SESSION_LOGIN = False +REST_USE_JWT = True ATOMIC_REQUESTS = False USE_X_FORWARDED_HOST = True @@ -428,6 +454,7 @@ ADMIN_URL = env('DJANGO_ADMIN_URL', default='^api/admin/') CSRF_USE_SESSIONS = True # Playlist settings +# XXX: deprecated, see #186 PLAYLISTS_MAX_TRACKS = env.int('PLAYLISTS_MAX_TRACKS', default=250) ACCOUNT_USERNAME_BLACKLIST = [ @@ -447,6 +474,8 @@ EXTERNAL_REQUESTS_VERIFY_SSL = env.bool( 'EXTERNAL_REQUESTS_VERIFY_SSL', default=True ) +# XXX: deprecated, see #186 +API_AUTHENTICATION_REQUIRED = env.bool("API_AUTHENTICATION_REQUIRED", True) MUSIC_DIRECTORY_PATH = env('MUSIC_DIRECTORY_PATH', default=None) # on Docker setup, the music directory may not match the host path, diff --git a/api/config/settings/local.py b/api/config/settings/local.py index dcbea66d2..592600629 100644 --- a/api/config/settings/local.py +++ b/api/config/settings/local.py @@ -25,9 +25,6 @@ SECRET_KEY = env("DJANGO_SECRET_KEY", default='mc$&b=5j#6^bv7tld1gyjp2&+^-qrdy=0 # ------------------------------------------------------------------------------ EMAIL_HOST = 'localhost' EMAIL_PORT = 1025 -EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', - default='django.core.mail.backends.console.EmailBackend') - # django-debug-toolbar # ------------------------------------------------------------------------------ diff --git a/api/config/settings/production.py b/api/config/settings/production.py index f238c2d20..2866e9103 100644 --- a/api/config/settings/production.py +++ b/api/config/settings/production.py @@ -68,16 +68,6 @@ DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' # ------------------------ STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' - -# EMAIL -# ------------------------------------------------------------------------------ -DEFAULT_FROM_EMAIL = env('DJANGO_DEFAULT_FROM_EMAIL', - default='funkwhale_api ') - -EMAIL_SUBJECT_PREFIX = env("DJANGO_EMAIL_SUBJECT_PREFIX", default='[funkwhale_api] ') -SERVER_EMAIL = env('DJANGO_SERVER_EMAIL', default=DEFAULT_FROM_EMAIL) - - # TEMPLATE CONFIGURATION # ------------------------------------------------------------------------------ # See: diff --git a/api/funkwhale_api/__init__.py b/api/funkwhale_api/__init__.py index 596926919..4f62dd9b5 100644 --- a/api/funkwhale_api/__init__.py +++ b/api/funkwhale_api/__init__.py @@ -1,3 +1,3 @@ # -*- coding: utf-8 -*- -__version__ = '0.10' +__version__ = '0.11' __version_info__ = tuple([int(num) if num.isdigit() else num for num in __version__.replace('-', '.', 1).split('.')]) diff --git a/api/funkwhale_api/common/auth.py b/api/funkwhale_api/common/auth.py index 75839b936..faf13571d 100644 --- a/api/funkwhale_api/common/auth.py +++ b/api/funkwhale_api/common/auth.py @@ -29,9 +29,6 @@ class TokenHeaderAuth(BaseJSONWebTokenAuthentication): class TokenAuthMiddleware: - """ - Custom middleware (insecure) that takes user IDs from the query string. - """ def __init__(self, inner): # Store the ASGI application we were passed diff --git a/api/funkwhale_api/common/authentication.py b/api/funkwhale_api/common/authentication.py index b75f3b516..c7566eac8 100644 --- a/api/funkwhale_api/common/authentication.py +++ b/api/funkwhale_api/common/authentication.py @@ -1,3 +1,6 @@ +from django.utils.encoding import smart_text +from django.utils.translation import ugettext as _ + from rest_framework import exceptions from rest_framework_jwt import authentication from rest_framework_jwt.settings import api_settings @@ -18,3 +21,37 @@ class JSONWebTokenAuthenticationQS( def authenticate_header(self, request): return '{0} realm="{1}"'.format( api_settings.JWT_AUTH_HEADER_PREFIX, self.www_authenticate_realm) + + +class BearerTokenHeaderAuth( + authentication.BaseJSONWebTokenAuthentication): + """ + For backward compatibility purpose, we used Authorization: JWT + but Authorization: Bearer is probably better. + """ + www_authenticate_realm = 'api' + + def get_jwt_value(self, request): + auth = authentication.get_authorization_header(request).split() + auth_header_prefix = 'bearer' + + if not auth: + if api_settings.JWT_AUTH_COOKIE: + return request.COOKIES.get(api_settings.JWT_AUTH_COOKIE) + return None + + if smart_text(auth[0].lower()) != auth_header_prefix: + return None + + if len(auth) == 1: + msg = _('Invalid Authorization header. No credentials provided.') + raise exceptions.AuthenticationFailed(msg) + elif len(auth) > 2: + msg = _('Invalid Authorization header. Credentials string ' + 'should not contain spaces.') + raise exceptions.AuthenticationFailed(msg) + + return auth[1] + + def authenticate_header(self, request): + return '{0} realm="{1}"'.format('Bearer', self.www_authenticate_realm) diff --git a/api/funkwhale_api/common/dynamic_preferences_registry.py b/api/funkwhale_api/common/dynamic_preferences_registry.py new file mode 100644 index 000000000..2374de7c7 --- /dev/null +++ b/api/funkwhale_api/common/dynamic_preferences_registry.py @@ -0,0 +1,20 @@ +from dynamic_preferences import types +from dynamic_preferences.registries import global_preferences_registry + +from funkwhale_api.common import preferences + +common = types.Section('common') + + +@global_preferences_registry.register +class APIAutenticationRequired( + preferences.DefaultFromSettingMixin, types.BooleanPreference): + section = common + name = 'api_authentication_required' + verbose_name = 'API Requires authentication' + setting = 'API_AUTHENTICATION_REQUIRED' + help_text = ( + 'If disabled, anonymous users will be able to query the API' + 'and access music data (as well as other data exposed in the API ' + 'without specific permissions)' + ) diff --git a/api/funkwhale_api/common/permissions.py b/api/funkwhale_api/common/permissions.py index c99c275c1..cab4b699d 100644 --- a/api/funkwhale_api/common/permissions.py +++ b/api/funkwhale_api/common/permissions.py @@ -5,11 +5,13 @@ from django.http import Http404 from rest_framework.permissions import BasePermission, DjangoModelPermissions +from funkwhale_api.common import preferences + class ConditionalAuthentication(BasePermission): def has_permission(self, request, view): - if settings.API_AUTHENTICATION_REQUIRED: + if preferences.get('common__api_authentication_required'): return request.user and request.user.is_authenticated return True diff --git a/api/funkwhale_api/common/preferences.py b/api/funkwhale_api/common/preferences.py new file mode 100644 index 000000000..e6eb8beda --- /dev/null +++ b/api/funkwhale_api/common/preferences.py @@ -0,0 +1,12 @@ +from django.conf import settings +from dynamic_preferences.registries import global_preferences_registry + + +class DefaultFromSettingMixin(object): + def get_default(self): + return getattr(settings, self.setting) + + +def get(pref): + manager = global_preferences_registry.manager() + return manager[pref] diff --git a/api/funkwhale_api/federation/actors.py b/api/funkwhale_api/federation/actors.py index 380bb23c0..7a209b1ff 100644 --- a/api/funkwhale_api/federation/actors.py +++ b/api/funkwhale_api/federation/actors.py @@ -1,3 +1,4 @@ +import datetime import logging import uuid import xml @@ -11,6 +12,7 @@ from rest_framework.exceptions import PermissionDenied from dynamic_preferences.registries import global_preferences_registry +from funkwhale_api.common import preferences from funkwhale_api.common import session from funkwhale_api.common import utils as funkwhale_utils from funkwhale_api.music import models as music_models @@ -49,11 +51,20 @@ def get_actor_data(actor_url): def get_actor(actor_url): + try: + actor = models.Actor.objects.get(url=actor_url) + except models.Actor.DoesNotExist: + actor = None + fetch_delta = datetime.timedelta( + minutes=preferences.get('federation__actor_fetch_delay')) + if actor and actor.last_fetch_date > timezone.now() - fetch_delta: + # cache is hot, we can return as is + return actor data = get_actor_data(actor_url) serializer = serializers.ActorSerializer(data=data) serializer.is_valid(raise_exception=True) - return serializer.build() + return serializer.save(last_fetch_date=timezone.now()) class SystemActor(object): @@ -215,7 +226,7 @@ class LibraryActor(SystemActor): @property def manually_approves_followers(self): - return settings.FEDERATION_MUSIC_NEEDS_APPROVAL + return preferences.get('federation__music_needs_approval') @transaction.atomic def handle_create(self, ac, sender): diff --git a/api/funkwhale_api/federation/authentication.py b/api/funkwhale_api/federation/authentication.py index 7f8ad6653..bfd46084c 100644 --- a/api/funkwhale_api/federation/authentication.py +++ b/api/funkwhale_api/federation/authentication.py @@ -25,28 +25,19 @@ class SignatureAuthentication(authentication.BaseAuthentication): raise exceptions.AuthenticationFailed(str(e)) try: - actor_data = actors.get_actor_data(key_id) + actor = actors.get_actor(key_id.split('#')[0]) except Exception as e: raise exceptions.AuthenticationFailed(str(e)) - try: - public_key = actor_data['publicKey']['publicKeyPem'] - except KeyError: + if not actor.public_key: raise exceptions.AuthenticationFailed('No public key found') - serializer = serializers.ActorSerializer(data=actor_data) - if not serializer.is_valid(): - raise exceptions.AuthenticationFailed('Invalid actor payload: {}'.format(serializer.errors)) - try: - signing.verify_django(request, public_key.encode('utf-8')) + signing.verify_django(request, actor.public_key.encode('utf-8')) except cryptography.exceptions.InvalidSignature: raise exceptions.AuthenticationFailed('Invalid signature') - try: - return models.Actor.objects.get(url=actor_data['id']) - except models.Actor.DoesNotExist: - return serializer.save() + return actor def authenticate(self, request): setattr(request, 'actor', None) diff --git a/api/funkwhale_api/federation/dynamic_preferences_registry.py b/api/funkwhale_api/federation/dynamic_preferences_registry.py index 43877c75c..e86b9f6f2 100644 --- a/api/funkwhale_api/federation/dynamic_preferences_registry.py +++ b/api/funkwhale_api/federation/dynamic_preferences_registry.py @@ -3,6 +3,7 @@ from django.forms import widgets from dynamic_preferences import types from dynamic_preferences.registries import global_preferences_registry +from funkwhale_api.common import preferences federation = types.Section('federation') @@ -18,3 +19,53 @@ class MusicCacheDuration(types.IntPreference): 'locally? Federated files that were not listened in this interval ' 'will be erased and refetched from the remote on the next listening.' ) + + +@global_preferences_registry.register +class Enabled(preferences.DefaultFromSettingMixin, types.BooleanPreference): + section = federation + name = 'enabled' + setting = 'FEDERATION_ENABLED' + verbose_name = 'Federation enabled' + help_text = ( + 'Use this setting to enable or disable federation logic and API' + ' globally' + ) + + +@global_preferences_registry.register +class CollectionPageSize( + preferences.DefaultFromSettingMixin, types.IntPreference): + section = federation + name = 'collection_page_size' + setting = 'FEDERATION_COLLECTION_PAGE_SIZE' + verbose_name = 'Federation collection page size' + help_text = ( + 'How much items to display in ActivityPub collections' + ) + + +@global_preferences_registry.register +class ActorFetchDelay( + preferences.DefaultFromSettingMixin, types.IntPreference): + section = federation + name = 'actor_fetch_delay' + setting = 'FEDERATION_ACTOR_FETCH_DELAY' + verbose_name = 'Federation actor fetch delay' + help_text = ( + 'How much minutes to wait before refetching actors on ' + 'request authentication' + ) + + +@global_preferences_registry.register +class MusicNeedsApproval( + preferences.DefaultFromSettingMixin, types.BooleanPreference): + section = federation + name = 'music_needs_approval' + setting = 'FEDERATION_MUSIC_NEEDS_APPROVAL' + verbose_name = 'Federation music needs approval' + help_text = ( + 'When true, other federation actors will require your approval' + ' before being able to browse your library.' + ) diff --git a/api/funkwhale_api/federation/permissions.py b/api/funkwhale_api/federation/permissions.py index c6f0660b1..438b675cb 100644 --- a/api/funkwhale_api/federation/permissions.py +++ b/api/funkwhale_api/federation/permissions.py @@ -2,13 +2,14 @@ from django.conf import settings from rest_framework.permissions import BasePermission +from funkwhale_api.common import preferences from . import actors class LibraryFollower(BasePermission): def has_permission(self, request, view): - if not settings.FEDERATION_MUSIC_NEEDS_APPROVAL: + if not preferences.get('federation__music_needs_approval'): return True actor = getattr(request, 'actor', None) diff --git a/api/funkwhale_api/federation/serializers.py b/api/funkwhale_api/federation/serializers.py index 00bb7d45b..426aabd77 100644 --- a/api/funkwhale_api/federation/serializers.py +++ b/api/funkwhale_api/federation/serializers.py @@ -1,3 +1,4 @@ +import logging import urllib.parse from django.urls import reverse @@ -21,6 +22,8 @@ AP_CONTEXT = [ {}, ] +logger = logging.getLogger(__name__) + class ActorSerializer(serializers.Serializer): id = serializers.URLField() @@ -100,9 +103,10 @@ class ActorSerializer(serializers.Serializer): def save(self, **kwargs): d = self.prepare_missing_fields() d.update(kwargs) - return models.Actor.objects.create( - **d - ) + return models.Actor.objects.update_or_create( + url=d['url'], + defaults=d, + )[0] def validate_summary(self, value): if value: @@ -620,6 +624,8 @@ class CollectionPageSerializer(serializers.Serializer): for i in raw_items: if i.is_valid(): valid_items.append(i) + else: + logger.debug('Invalid item %s: %s', i.data, i.errors) return valid_items diff --git a/api/funkwhale_api/federation/tasks.py b/api/funkwhale_api/federation/tasks.py index adc354c4f..8f931b0ed 100644 --- a/api/funkwhale_api/federation/tasks.py +++ b/api/funkwhale_api/federation/tasks.py @@ -1,8 +1,10 @@ import datetime import json import logging +import os from django.conf import settings +from django.db.models import Q from django.utils import timezone from requests.exceptions import RequestException @@ -96,16 +98,38 @@ def clean_music_cache(): delay = preferences['federation__music_cache_duration'] if delay < 1: return # cache clearing disabled + limit = timezone.now() - datetime.timedelta(minutes=delay) candidates = models.LibraryTrack.objects.filter( - audio_file__isnull=False - ).values_list('local_track_file__track', flat=True) - listenings = Listening.objects.filter( - creation_date__gte=timezone.now() - datetime.timedelta(minutes=delay), - track__pk__in=candidates).values_list('track', flat=True) - too_old = set(candidates) - set(listenings) - - to_remove = models.LibraryTrack.objects.filter( - local_track_file__track__pk__in=too_old).only('audio_file') - for lt in to_remove: + Q(audio_file__isnull=False) & ( + Q(local_track_file__accessed_date__lt=limit) | + Q(local_track_file__accessed_date=None) + ) + ).exclude(audio_file='').only('audio_file', 'id') + for lt in candidates: lt.audio_file.delete() + + # we also delete orphaned files, if any + storage = models.LibraryTrack._meta.get_field('audio_file').storage + files = get_files(storage, 'federation_cache') + existing = models.LibraryTrack.objects.filter(audio_file__in=files) + missing = set(files) - set(existing.values_list('audio_file', flat=True)) + for m in missing: + storage.delete(m) + + +def get_files(storage, *parts): + """ + This is a recursive function that return all files available + in a given directory using django's storage. + """ + if not parts: + raise ValueError('Missing path') + + dirs, files = storage.listdir(os.path.join(*parts)) + for dir in dirs: + files += get_files(storage, *(list(parts) + [dir])) + return [ + os.path.join(parts[-1], path) + for path in files + ] diff --git a/api/funkwhale_api/federation/views.py b/api/funkwhale_api/federation/views.py index 381f87eff..9b51a534d 100644 --- a/api/funkwhale_api/federation/views.py +++ b/api/funkwhale_api/federation/views.py @@ -13,6 +13,7 @@ from rest_framework import viewsets from rest_framework.decorators import list_route, detail_route from rest_framework.serializers import ValidationError +from funkwhale_api.common import preferences from funkwhale_api.common import utils as funkwhale_utils from funkwhale_api.common.permissions import HasModelPermission from funkwhale_api.music.models import TrackFile @@ -33,7 +34,7 @@ from . import webfinger class FederationMixin(object): def dispatch(self, request, *args, **kwargs): - if not settings.FEDERATION_ENABLED: + if not preferences.get('federation__enabled'): return HttpResponse(status=405) return super().dispatch(request, *args, **kwargs) @@ -136,7 +137,8 @@ class MusicFilesViewSet(FederationMixin, viewsets.GenericViewSet): if page is None: conf = { 'id': utils.full_url(reverse('federation:music:files-list')), - 'page_size': settings.FEDERATION_COLLECTION_PAGE_SIZE, + 'page_size': preferences.get( + 'federation__collection_page_size'), 'items': qs, 'item_serializer': serializers.AudioSerializer, 'actor': library, @@ -150,7 +152,7 @@ class MusicFilesViewSet(FederationMixin, viewsets.GenericViewSet): return response.Response( {'page': ['Invalid page number']}, status=400) p = paginator.Paginator( - qs, settings.FEDERATION_COLLECTION_PAGE_SIZE) + qs, preferences.get('federation__collection_page_size')) try: page = p.page(page_number) conf = { diff --git a/api/funkwhale_api/history/models.py b/api/funkwhale_api/history/models.py index 762d5bf7b..480461d35 100644 --- a/api/funkwhale_api/history/models.py +++ b/api/funkwhale_api/history/models.py @@ -21,13 +21,6 @@ class Listening(models.Model): class Meta: ordering = ('-creation_date',) - def save(self, **kwargs): - if not self.user and not self.session_key: - raise ValidationError('Cannot have both session_key and user empty for listening') - - super().save(**kwargs) - - def get_activity_url(self): return '{}/listenings/tracks/{}'.format( self.user.get_activity_url(), self.pk) diff --git a/api/funkwhale_api/history/serializers.py b/api/funkwhale_api/history/serializers.py index 8fe6fa6e0..f7333f243 100644 --- a/api/funkwhale_api/history/serializers.py +++ b/api/funkwhale_api/history/serializers.py @@ -36,13 +36,9 @@ class ListeningSerializer(serializers.ModelSerializer): class Meta: model = models.Listening - fields = ('id', 'user', 'session_key', 'track', 'creation_date') - + fields = ('id', 'user', 'track', 'creation_date') def create(self, validated_data): - if self.context.get('user'): - validated_data['user'] = self.context.get('user') - else: - validated_data['session_key'] = self.context['session_key'] + validated_data['user'] = self.context['user'] return super().create(validated_data) diff --git a/api/funkwhale_api/history/views.py b/api/funkwhale_api/history/views.py index d5cbe316b..bea96a418 100644 --- a/api/funkwhale_api/history/views.py +++ b/api/funkwhale_api/history/views.py @@ -1,4 +1,5 @@ from rest_framework import generics, mixins, viewsets +from rest_framework import permissions from rest_framework import status from rest_framework.response import Response from rest_framework.decorators import detail_route @@ -10,31 +11,26 @@ from funkwhale_api.music.serializers import TrackSerializerNested from . import models from . import serializers -class ListeningViewSet(mixins.CreateModelMixin, - mixins.RetrieveModelMixin, - viewsets.GenericViewSet): + +class ListeningViewSet( + mixins.CreateModelMixin, + mixins.RetrieveModelMixin, + viewsets.GenericViewSet): serializer_class = serializers.ListeningSerializer queryset = models.Listening.objects.all() - permission_classes = [ConditionalAuthentication] + permission_classes = [permissions.IsAuthenticated] def perform_create(self, serializer): r = super().perform_create(serializer) - if self.request.user.is_authenticated: - record.send(serializer.instance) + record.send(serializer.instance) return r def get_queryset(self): queryset = super().get_queryset() - if self.request.user.is_authenticated: - return queryset.filter(user=self.request.user) - else: - return queryset.filter(session_key=self.request.session.session_key) + return queryset.filter(user=self.request.user) def get_serializer_context(self): context = super().get_serializer_context() - if self.request.user.is_authenticated: - context['user'] = self.request.user - else: - context['session_key'] = self.request.session.session_key + context['user'] = self.request.user return context diff --git a/api/funkwhale_api/music/filters.py b/api/funkwhale_api/music/filters.py index 752422e75..6da9cca63 100644 --- a/api/funkwhale_api/music/filters.py +++ b/api/funkwhale_api/music/filters.py @@ -20,6 +20,9 @@ class ListenableMixin(filters.FilterSet): class ArtistFilter(ListenableMixin): + q = fields.SearchFilter(search_fields=[ + 'name', + ]) class Meta: model = models.Artist diff --git a/api/funkwhale_api/music/management/__init__.py b/api/funkwhale_api/music/management/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/api/funkwhale_api/music/management/commands/__init__.py b/api/funkwhale_api/music/management/commands/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/api/funkwhale_api/music/management/commands/fix_track_files.py b/api/funkwhale_api/music/management/commands/fix_track_files.py new file mode 100644 index 000000000..f68bcf135 --- /dev/null +++ b/api/funkwhale_api/music/management/commands/fix_track_files.py @@ -0,0 +1,45 @@ +import cacheops +import os + +from django.db import transaction +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError + +from funkwhale_api.music import models, utils + + +class Command(BaseCommand): + help = 'Run common checks and fix against imported tracks' + + def add_arguments(self, parser): + parser.add_argument( + '--dry-run', + action='store_true', + dest='dry_run', + default=False, + help='Do not execute anything' + ) + + def handle(self, *args, **options): + if options['dry_run']: + self.stdout.write('Dry-run on, will not commit anything') + self.fix_mimetypes(**options) + cacheops.invalidate_model(models.TrackFile) + + @transaction.atomic + def fix_mimetypes(self, dry_run, **kwargs): + self.stdout.write('Fixing missing mimetypes...') + matching = models.TrackFile.objects.filter( + source__startswith='file://', mimetype=None) + self.stdout.write( + '[mimetypes] {} entries found with no mimetype'.format( + matching.count())) + for extension, mimetype in utils.EXTENSION_TO_MIMETYPE.items(): + qs = matching.filter(source__endswith='.{}'.format(extension)) + self.stdout.write( + '[mimetypes] setting {} {} files to {}'.format( + qs.count(), extension, mimetype + )) + if not dry_run: + self.stdout.write('[mimetypes] commiting...') + qs.update(mimetype=mimetype) diff --git a/api/funkwhale_api/music/migrations/0026_trackfile_accessed_date.py b/api/funkwhale_api/music/migrations/0026_trackfile_accessed_date.py new file mode 100644 index 000000000..1d5327d93 --- /dev/null +++ b/api/funkwhale_api/music/migrations/0026_trackfile_accessed_date.py @@ -0,0 +1,18 @@ +# Generated by Django 2.0.3 on 2018-05-06 12:47 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('music', '0025_auto_20180419_2023'), + ] + + operations = [ + migrations.AddField( + model_name='trackfile', + name='accessed_date', + field=models.DateTimeField(blank=True, null=True), + ), + ] diff --git a/api/funkwhale_api/music/models.py b/api/funkwhale_api/music/models.py index 98fc1965b..655d38755 100644 --- a/api/funkwhale_api/music/models.py +++ b/api/funkwhale_api/music/models.py @@ -415,6 +415,7 @@ class TrackFile(models.Model): source = models.URLField(null=True, blank=True, max_length=500) creation_date = models.DateTimeField(default=timezone.now) modification_date = models.DateTimeField(auto_now=True) + accessed_date = models.DateTimeField(null=True, blank=True) duration = models.IntegerField(null=True, blank=True) acoustid_track_id = models.UUIDField(null=True, blank=True) mimetype = models.CharField(null=True, blank=True, max_length=200) @@ -463,26 +464,6 @@ class TrackFile(models.Model): self.mimetype = utils.guess_mimetype(self.audio_file) return super().save(**kwargs) - @property - def serve_from_source_path(self): - if not self.source or not self.source.startswith('file://'): - raise ValueError('Cannot serve this file from source') - serve_path = settings.MUSIC_DIRECTORY_SERVE_PATH - prefix = settings.MUSIC_DIRECTORY_PATH - if not serve_path or not prefix: - raise ValueError( - 'You need to specify MUSIC_DIRECTORY_SERVE_PATH and ' - 'MUSIC_DIRECTORY_PATH to serve in-place imported files' - ) - file_path = self.source.replace('file://', '', 1) - parts = os.path.split(file_path.replace(prefix, '', 1)) - if parts[0] == '/': - parts = parts[1:] - return os.path.join( - serve_path, - *parts - ) - IMPORT_STATUS_CHOICES = ( ('pending', 'Pending'), diff --git a/api/funkwhale_api/music/permissions.py b/api/funkwhale_api/music/permissions.py index 61fc65beb..77f95c477 100644 --- a/api/funkwhale_api/music/permissions.py +++ b/api/funkwhale_api/music/permissions.py @@ -2,6 +2,7 @@ from django.conf import settings from rest_framework.permissions import BasePermission +from funkwhale_api.common import preferences from funkwhale_api.federation import actors from funkwhale_api.federation import models @@ -12,6 +13,9 @@ class Listen(BasePermission): if not settings.PROTECT_AUDIO_FILES: return True + if not preferences.get('common__api_authentication_required'): + return True + user = getattr(request, 'user', None) if user and user.is_authenticated: return True diff --git a/api/funkwhale_api/music/serializers.py b/api/funkwhale_api/music/serializers.py index b9ecfc50d..9dfc91478 100644 --- a/api/funkwhale_api/music/serializers.py +++ b/api/funkwhale_api/music/serializers.py @@ -1,4 +1,5 @@ from django.db import transaction +from django.db.models import Q from rest_framework import serializers from taggit.models import Tag @@ -9,6 +10,7 @@ from funkwhale_api.federation.serializers import AP_CONTEXT from funkwhale_api.users.serializers import UserBasicSerializer from . import models +from . import tasks class TagSerializer(serializers.ModelSerializer): @@ -204,3 +206,33 @@ class SubmitFederationTracksSerializer(serializers.Serializer): source=lt.url, ) return batch + + +class ImportJobRunSerializer(serializers.Serializer): + jobs = serializers.PrimaryKeyRelatedField( + many=True, + queryset=models.ImportJob.objects.filter( + status__in=['pending', 'errored'] + ) + ) + batches = serializers.PrimaryKeyRelatedField( + many=True, + queryset=models.ImportBatch.objects.all() + ) + + def validate(self, validated_data): + jobs = validated_data['jobs'] + batches_ids = [b.pk for b in validated_data['batches']] + query = Q(batch__pk__in=batches_ids) + query |= Q(pk__in=[j.id for j in jobs]) + queryset = models.ImportJob.objects.filter(query).filter( + status__in=['pending', 'errored']).distinct() + validated_data['_jobs'] = queryset + return validated_data + + def create(self, validated_data): + ids = validated_data['_jobs'].values_list('id', flat=True) + validated_data['_jobs'].update(status='pending') + for id in ids: + tasks.import_job_run.delay(import_job_id=id) + return {'jobs': list(ids)} diff --git a/api/funkwhale_api/music/tasks.py b/api/funkwhale_api/music/tasks.py index f2244d785..4509c9a57 100644 --- a/api/funkwhale_api/music/tasks.py +++ b/api/funkwhale_api/music/tasks.py @@ -1,7 +1,8 @@ +import os + from django.core.files.base import ContentFile -from dynamic_preferences.registries import global_preferences_registry - +from funkwhale_api.common import preferences from funkwhale_api.federation import activity from funkwhale_api.federation import actors from funkwhale_api.federation import models as federation_models @@ -13,6 +14,7 @@ from funkwhale_api.providers.audiofile.tasks import import_track_data_from_path from django.conf import settings from . import models from . import lyrics as lyrics_utils +from . import utils as music_utils @celery.app.task(name='acoustid.set_on_track_file') @@ -77,8 +79,7 @@ def _do_import(import_job, replace=False, use_acoustid=True): acoustid_track_id = None duration = None track = None - manager = global_preferences_registry.manager() - use_acoustid = use_acoustid and manager['providers_acoustid__api_key'] + use_acoustid = use_acoustid and preferences.get('providers_acoustid__api_key') if not mbid and use_acoustid and from_file: # we try to deduce mbid from acoustid client = get_acoustid_client() @@ -129,6 +130,10 @@ def _do_import(import_job, replace=False, use_acoustid=True): elif not import_job.audio_file and not import_job.source.startswith('file://'): # not an implace import, and we have a source, so let's download it track_file.download_file() + elif not import_job.audio_file and import_job.source.startswith('file://'): + # in place import, we set mimetype from extension + path, ext = os.path.splitext(import_job.source) + track_file.mimetype = music_utils.get_type_from_ext(ext) track_file.save() import_job.status = 'finished' import_job.track_file = track_file @@ -178,7 +183,7 @@ def fetch_content(lyrics): @celery.require_instance( models.ImportBatch.objects.filter(status='finished'), 'import_batch') def import_batch_notify_followers(import_batch): - if not settings.FEDERATION_ENABLED: + if not preferences.get('federation__enabled'): return if import_batch.source == 'federation': diff --git a/api/funkwhale_api/music/utils.py b/api/funkwhale_api/music/utils.py index 7a851f7cc..49a639303 100644 --- a/api/funkwhale_api/music/utils.py +++ b/api/funkwhale_api/music/utils.py @@ -63,8 +63,21 @@ def compute_status(jobs): return 'finished' +AUDIO_EXTENSIONS_AND_MIMETYPE = [ + ('ogg', 'audio/ogg'), + ('mp3', 'audio/mpeg'), +] + +EXTENSION_TO_MIMETYPE = {ext: mt for ext, mt in AUDIO_EXTENSIONS_AND_MIMETYPE} +MIMETYPE_TO_EXTENSION = {mt: ext for ext, mt in AUDIO_EXTENSIONS_AND_MIMETYPE} + + def get_ext_from_type(mimetype): - mapping = { - 'audio/ogg': 'ogg', - 'audio/mpeg': 'mp3', - } + return MIMETYPE_TO_EXTENSION.get(mimetype) + + +def get_type_from_ext(extension): + if extension.startswith('.'): + # we remove leading dot + extension = extension[1:] + return EXTENSION_TO_MIMETYPE.get(extension) diff --git a/api/funkwhale_api/music/views.py b/api/funkwhale_api/music/views.py index af063da46..76fc8bc3e 100644 --- a/api/funkwhale_api/music/views.py +++ b/api/funkwhale_api/music/views.py @@ -14,6 +14,7 @@ from django.db.models.functions import Length from django.db.models import Count from django.http import StreamingHttpResponse from django.urls import reverse +from django.utils import timezone from django.utils.decorators import method_decorator from rest_framework import viewsets, views, mixins @@ -145,6 +146,14 @@ class ImportJobViewSet( data['count'] = sum([v for v in data.values()]) return Response(data) + @list_route(methods=['post']) + def run(self, request, *args, **kwargs): + serializer = serializers.ImportJobRunSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + payload = serializer.save() + + return Response(payload) + def perform_create(self, serializer): source = 'file://' + serializer.validated_data['audio_file'].name serializer.save(source=source) @@ -206,6 +215,8 @@ class TrackViewSet( def get_file_path(audio_file): + serve_path = settings.MUSIC_DIRECTORY_SERVE_PATH + prefix = settings.MUSIC_DIRECTORY_PATH t = settings.REVERSE_PROXY_TYPE if t == 'nginx': # we have to use the internal locations @@ -213,14 +224,24 @@ def get_file_path(audio_file): path = audio_file.url except AttributeError: # a path was given - path = '/music' + audio_file + if not serve_path or not prefix: + raise ValueError( + 'You need to specify MUSIC_DIRECTORY_SERVE_PATH and ' + 'MUSIC_DIRECTORY_PATH to serve in-place imported files' + ) + path = '/music' + audio_file.replace(prefix, '', 1) return settings.PROTECT_FILES_PATH + path if t == 'apache2': try: path = audio_file.path except AttributeError: # a path was given - path = audio_file + if not serve_path or not prefix: + raise ValueError( + 'You need to specify MUSIC_DIRECTORY_SERVE_PATH and ' + 'MUSIC_DIRECTORY_PATH to serve in-place imported files' + ) + path = audio_file.replace(prefix, serve_path, 1) return path @@ -244,6 +265,10 @@ class TrackFileViewSet(viewsets.ReadOnlyModelViewSet): except models.TrackFile.DoesNotExist: return Response(status=404) + # we update the accessed_date + f.accessed_date = timezone.now() + f.save(update_fields=['accessed_date']) + mt = f.mimetype audio_file = f.audio_file try: @@ -267,7 +292,7 @@ class TrackFileViewSet(viewsets.ReadOnlyModelViewSet): elif audio_file: file_path = get_file_path(audio_file) elif f.source and f.source.startswith('file://'): - file_path = get_file_path(f.serve_from_source_path) + file_path = get_file_path(f.source.replace('file://', '', 1)) response = Response() filename = f.filename mapping = { diff --git a/api/funkwhale_api/playlists/dynamic_preferences_registry.py b/api/funkwhale_api/playlists/dynamic_preferences_registry.py new file mode 100644 index 000000000..21140fa14 --- /dev/null +++ b/api/funkwhale_api/playlists/dynamic_preferences_registry.py @@ -0,0 +1,15 @@ +from dynamic_preferences import types +from dynamic_preferences.registries import global_preferences_registry + +from funkwhale_api.common import preferences + +playlists = types.Section('playlists') + + +@global_preferences_registry.register +class MaxTracks(preferences.DefaultFromSettingMixin, types.IntegerPreference): + show_in_api = True + section = playlists + name = 'max_tracks' + verbose_name = 'Max tracks per playlist' + setting = 'PLAYLISTS_MAX_TRACKS' diff --git a/api/funkwhale_api/playlists/models.py b/api/funkwhale_api/playlists/models.py index 6bb8fe178..a208a5fd0 100644 --- a/api/funkwhale_api/playlists/models.py +++ b/api/funkwhale_api/playlists/models.py @@ -6,6 +6,7 @@ from django.utils import timezone from rest_framework import exceptions from funkwhale_api.common import fields +from funkwhale_api.common import preferences class Playlist(models.Model): @@ -81,10 +82,11 @@ class Playlist(models.Model): existing = self.playlist_tracks.select_for_update() now = timezone.now() total = existing.filter(index__isnull=False).count() - if existing.count() + len(tracks) > settings.PLAYLISTS_MAX_TRACKS: + max_tracks = preferences.get('playlists__max_tracks') + if existing.count() + len(tracks) > max_tracks: raise exceptions.ValidationError( 'Playlist would reach the maximum of {} tracks'.format( - settings.PLAYLISTS_MAX_TRACKS)) + max_tracks)) self.save(update_fields=['modification_date']) start = total plts = [ diff --git a/api/funkwhale_api/playlists/serializers.py b/api/funkwhale_api/playlists/serializers.py index 6caf9aa4a..fcb2a412d 100644 --- a/api/funkwhale_api/playlists/serializers.py +++ b/api/funkwhale_api/playlists/serializers.py @@ -3,6 +3,7 @@ from django.db import transaction from rest_framework import serializers from taggit.models import Tag +from funkwhale_api.common import preferences from funkwhale_api.music.models import Track from funkwhale_api.music.serializers import TrackSerializerNested from funkwhale_api.users.serializers import UserBasicSerializer @@ -32,10 +33,11 @@ class PlaylistTrackWriteSerializer(serializers.ModelSerializer): raise serializers.ValidationError( 'You do not have the permission to edit this playlist') existing = value.playlist_tracks.count() - if existing >= settings.PLAYLISTS_MAX_TRACKS: + max_tracks = preferences.get('playlists__max_tracks') + if existing >= max_tracks: raise serializers.ValidationError( 'Playlist has reached the maximum of {} tracks'.format( - settings.PLAYLISTS_MAX_TRACKS)) + max_tracks)) return value @transaction.atomic diff --git a/api/funkwhale_api/radios/models.py b/api/funkwhale_api/radios/models.py index 0273b5387..8758abc61 100644 --- a/api/funkwhale_api/radios/models.py +++ b/api/funkwhale_api/radios/models.py @@ -55,8 +55,6 @@ class RadioSession(models.Model): related_object = GenericForeignKey('related_object_content_type', 'related_object_id') def save(self, **kwargs): - if not self.user and not self.session_key: - raise ValidationError('Cannot have both session_key and user empty for radio session') self.radio.clean(self) super().save(**kwargs) diff --git a/api/funkwhale_api/radios/serializers.py b/api/funkwhale_api/radios/serializers.py index 2e7e6a409..195b382c9 100644 --- a/api/funkwhale_api/radios/serializers.py +++ b/api/funkwhale_api/radios/serializers.py @@ -38,6 +38,7 @@ class RadioSerializer(serializers.ModelSerializer): return super().save(**kwargs) + class RadioSessionTrackSerializerCreate(serializers.ModelSerializer): class Meta: model = models.RadioSessionTrack @@ -62,17 +63,14 @@ class RadioSessionSerializer(serializers.ModelSerializer): 'user', 'creation_date', 'custom_radio', - 'session_key') + ) def validate(self, data): registry[data['radio_type']]().validate_session(data, **self.context) return data def create(self, validated_data): - if self.context.get('user'): - validated_data['user'] = self.context.get('user') - else: - validated_data['session_key'] = self.context['session_key'] + validated_data['user'] = self.context['user'] if validated_data.get('related_object_id'): radio = registry[validated_data['radio_type']]() validated_data['related_object'] = radio.get_related_object(validated_data['related_object_id']) diff --git a/api/funkwhale_api/radios/views.py b/api/funkwhale_api/radios/views.py index ffd1d1659..37c07c5e4 100644 --- a/api/funkwhale_api/radios/views.py +++ b/api/funkwhale_api/radios/views.py @@ -2,6 +2,7 @@ from django.db.models import Q from django.http import Http404 from rest_framework import generics, mixins, viewsets +from rest_framework import permissions from rest_framework import status from rest_framework.response import Response from rest_framework.decorators import detail_route, list_route @@ -24,7 +25,7 @@ class RadioViewSet( viewsets.GenericViewSet): serializer_class = serializers.RadioSerializer - permission_classes = [ConditionalAuthentication] + permission_classes = [permissions.IsAuthenticated] filter_class = filtersets.RadioFilter def get_queryset(self): @@ -84,21 +85,15 @@ class RadioSessionViewSet(mixins.CreateModelMixin, serializer_class = serializers.RadioSessionSerializer queryset = models.RadioSession.objects.all() - permission_classes = [ConditionalAuthentication] + permission_classes = [permissions.IsAuthenticated] def get_queryset(self): queryset = super().get_queryset() - if self.request.user.is_authenticated: - return queryset.filter(user=self.request.user) - else: - return queryset.filter(session_key=self.request.session.session_key) + return queryset.filter(user=self.request.user) def get_serializer_context(self): context = super().get_serializer_context() - if self.request.user.is_authenticated: - context['user'] = self.request.user - else: - context['session_key'] = self.request.session.session_key + context['user'] = self.request.user return context @@ -106,17 +101,14 @@ class RadioSessionTrackViewSet(mixins.CreateModelMixin, viewsets.GenericViewSet): serializer_class = serializers.RadioSessionTrackSerializer queryset = models.RadioSessionTrack.objects.all() - permission_classes = [ConditionalAuthentication] + permission_classes = [permissions.IsAuthenticated] def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) serializer.is_valid(raise_exception=True) session = serializer.validated_data['session'] try: - if request.user.is_authenticated: - assert request.user == session.user - else: - assert request.session.session_key == session.session_key + assert request.user == session.user except AssertionError: return Response(status=status.HTTP_403_FORBIDDEN) track = session.radio.pick() diff --git a/api/funkwhale_api/requests/filters.py b/api/funkwhale_api/requests/filters.py index bf353e8ad..7d0603362 100644 --- a/api/funkwhale_api/requests/filters.py +++ b/api/funkwhale_api/requests/filters.py @@ -1,10 +1,18 @@ import django_filters +from funkwhale_api.common import fields from . import models class ImportRequestFilter(django_filters.FilterSet): + q = fields.SearchFilter(search_fields=[ + 'artist_name', + 'user__username', + 'albums', + 'comment', + ]) + class Meta: model = models.ImportRequest fields = { diff --git a/api/funkwhale_api/requests/models.py b/api/funkwhale_api/requests/models.py index c29852430..d08dd4004 100644 --- a/api/funkwhale_api/requests/models.py +++ b/api/funkwhale_api/requests/models.py @@ -15,6 +15,7 @@ STATUS_CHOICES = [ ('closed', 'closed'), ] + class ImportRequest(models.Model): creation_date = models.DateTimeField(default=timezone.now) imported_date = models.DateTimeField(null=True, blank=True) diff --git a/api/funkwhale_api/templates/account/email/email_confirmation_message.txt b/api/funkwhale_api/templates/account/email/email_confirmation_message.txt new file mode 100644 index 000000000..8aec540fe --- /dev/null +++ b/api/funkwhale_api/templates/account/email/email_confirmation_message.txt @@ -0,0 +1,8 @@ +{% load account %}{% user_display user as user_display %}{% load i18n %}{% autoescape off %}{% blocktrans with site_name=current_site.name site_domain=current_site.domain %}Hello from {{ site_name }}! + +You're receiving this e-mail because user {{ user_display }} at {{ site_domain }} has given yours as an e-mail address to connect their account. + +To confirm this is correct, go to {{ funkwhale_url }}/auth/email/confirm?key={{ key }} +{% endblocktrans %}{% endautoescape %} +{% blocktrans with site_name=current_site.name site_domain=current_site.domain %}Thank you from {{ site_name }}! +{{ site_domain }}{% endblocktrans %} diff --git a/api/funkwhale_api/templates/registration/password_reset_email.html b/api/funkwhale_api/templates/registration/password_reset_email.html new file mode 100644 index 000000000..7a587d720 --- /dev/null +++ b/api/funkwhale_api/templates/registration/password_reset_email.html @@ -0,0 +1,12 @@ +{% load i18n %}{% autoescape off %} +{% blocktrans %}You're receiving this email because you requested a password reset for your user account at {{ site_name }}.{% endblocktrans %} + +{% trans "Please go to the following page and choose a new password:" %} +{{ funkwhale_url }}/auth/password/reset/confirm?uid={{ uid }}&token={{ token }} +{% trans "Your username, in case you've forgotten:" %} {{ user.get_username }} + +{% trans "Thanks for using our site!" %} + +{% blocktrans %}The {{ site_name }} team{% endblocktrans %} + +{% endautoescape %} diff --git a/api/funkwhale_api/users/adapters.py b/api/funkwhale_api/users/adapters.py index 96d1b8b1d..7bd341d14 100644 --- a/api/funkwhale_api/users/adapters.py +++ b/api/funkwhale_api/users/adapters.py @@ -1,5 +1,6 @@ -from allauth.account.adapter import DefaultAccountAdapter +from django.conf import settings +from allauth.account.adapter import DefaultAccountAdapter from dynamic_preferences.registries import global_preferences_registry @@ -8,3 +9,7 @@ class FunkwhaleAccountAdapter(DefaultAccountAdapter): def is_open_for_signup(self, request): manager = global_preferences_registry.manager() return manager['users__registration_enabled'] + + def send_mail(self, template_prefix, email, context): + context['funkwhale_url'] = settings.FUNKWHALE_URL + return super().send_mail(template_prefix, email, context) diff --git a/api/funkwhale_api/users/middleware.py b/api/funkwhale_api/users/middleware.py deleted file mode 100644 index e3eba95f3..000000000 --- a/api/funkwhale_api/users/middleware.py +++ /dev/null @@ -1,11 +0,0 @@ - - -class AnonymousSessionMiddleware: - def __init__(self, get_response): - self.get_response = get_response - - def __call__(self, request): - if not request.session.session_key: - request.session.save() - response = self.get_response(request) - return response diff --git a/api/funkwhale_api/users/rest_auth_urls.py b/api/funkwhale_api/users/rest_auth_urls.py index 31f5384aa..fa6c425cc 100644 --- a/api/funkwhale_api/users/rest_auth_urls.py +++ b/api/funkwhale_api/users/rest_auth_urls.py @@ -1,16 +1,20 @@ from django.views.generic import TemplateView from django.conf.urls import url -from rest_auth.registration.views import VerifyEmailView -from rest_auth.views import PasswordChangeView +from rest_auth.registration import views as registration_views +from rest_auth import views as rest_auth_views -from .views import RegisterView +from . import views urlpatterns = [ - url(r'^$', RegisterView.as_view(), name='rest_register'), - url(r'^verify-email/$', VerifyEmailView.as_view(), name='rest_verify_email'), - url(r'^change-password/$', PasswordChangeView.as_view(), name='change_password'), + url(r'^$', views.RegisterView.as_view(), name='rest_register'), + url(r'^verify-email/$', + registration_views.VerifyEmailView.as_view(), + name='rest_verify_email'), + url(r'^change-password/$', + rest_auth_views.PasswordChangeView.as_view(), + name='change_password'), # This url is used by django-allauth and empty TemplateView is # defined just to allow reverse() call inside app, for example when email diff --git a/api/funkwhale_api/users/serializers.py b/api/funkwhale_api/users/serializers.py index b21aa6935..eadce6154 100644 --- a/api/funkwhale_api/users/serializers.py +++ b/api/funkwhale_api/users/serializers.py @@ -1,5 +1,7 @@ -from rest_framework import serializers +from django.conf import settings +from rest_framework import serializers +from rest_auth.serializers import PasswordResetSerializer as PRS from funkwhale_api.activity import serializers as activity_serializers from . import models @@ -63,3 +65,12 @@ class UserReadSerializer(serializers.ModelSerializer): 'status': o.has_perm(internal_codename) } return perms + + +class PasswordResetSerializer(PRS): + def get_email_options(self): + return { + 'extra_email_context': { + 'funkwhale_url': settings.FUNKWHALE_URL + } + } diff --git a/api/setup.cfg b/api/setup.cfg index a2b8b92c6..b1267c904 100644 --- a/api/setup.cfg +++ b/api/setup.cfg @@ -11,7 +11,7 @@ python_files = tests.py test_*.py *_tests.py testpaths = tests env = SECRET_KEY=test - DJANGO_EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend + EMAIL_CONFIG=consolemail:// CELERY_BROKER_URL=memory:// CELERY_TASK_ALWAYS_EAGER=True CACHEOPS_ENABLED=False diff --git a/api/tests/activity/test_views.py b/api/tests/activity/test_views.py index bdc3c6339..9b24f3ad3 100644 --- a/api/tests/activity/test_views.py +++ b/api/tests/activity/test_views.py @@ -4,8 +4,8 @@ from funkwhale_api.activity import serializers from funkwhale_api.activity import utils -def test_activity_view(factories, api_client, settings, anonymous_user): - settings.API_AUTHENTICATION_REQUIRED = False +def test_activity_view(factories, api_client, preferences, anonymous_user): + preferences['common__api_authentication_required'] = False favorite = factories['favorites.TrackFavorite']( user__privacy_level='everyone') listening = factories['history.Listening']() diff --git a/api/tests/conftest.py b/api/tests/conftest.py index 64dc394e7..51a1bc4c7 100644 --- a/api/tests/conftest.py +++ b/api/tests/conftest.py @@ -27,12 +27,19 @@ def factories_autodiscover(): @pytest.fixture(autouse=True) def cache(): + """ + Returns a django Cache instance for cache-related operations + """ yield django_cache django_cache.clear() @pytest.fixture def factories(db): + """ + Returns a dictionnary containing all registered factories with keys such as + users.User or music.Track + """ from funkwhale_api import factories for v in factories.registry.values(): try: @@ -45,6 +52,10 @@ def factories(db): @pytest.fixture def nodb_factories(): + """ + Returns a dictionnary containing all registered factories with a build strategy + that does not require access to the database + """ from funkwhale_api import factories for v in factories.registry.values(): try: @@ -57,6 +68,9 @@ def nodb_factories(): @pytest.fixture def preferences(db, cache): + """ + return a dynamic_preferences manager for global_preferences + """ manager = global_preferences_registry.manager() manager.all() yield manager @@ -64,6 +78,10 @@ def preferences(db, cache): @pytest.fixture def tmpdir(): + """ + Returns a temporary directory path where you can write things during your + test + """ d = tempfile.mkdtemp() yield d shutil.rmtree(d) @@ -71,11 +89,18 @@ def tmpdir(): @pytest.fixture def tmpfile(): + """ + Returns a temporary file where you can write things during your test + """ yield tempfile.NamedTemporaryFile() @pytest.fixture def logged_in_client(db, factories, client): + """ + Returns a logged-in, non-API client with an authenticated ``User`` + stored in the ``user`` attribute + """ user = factories['users.User']() assert client.login(username=user.username, password='test') setattr(client, 'user', user) @@ -85,16 +110,24 @@ def logged_in_client(db, factories, client): @pytest.fixture def anonymous_user(): + """Returns a AnonymousUser() instance""" return AnonymousUser() @pytest.fixture def api_client(client): + """ + Return an API client without any authentication + """ return APIClient() @pytest.fixture def logged_in_api_client(db, factories, api_client): + """ + Return a logged-in API client with an authenticated ``User`` + stored in the ``user`` attribute + """ user = factories['users.User']() assert api_client.login(username=user.username, password='test') setattr(api_client, 'user', user) @@ -104,6 +137,10 @@ def logged_in_api_client(db, factories, api_client): @pytest.fixture def superuser_api_client(db, factories, api_client): + """ + Return a logged-in API client with an authenticated superuser + stored in the ``user`` attribute + """ user = factories['users.SuperUser']() assert api_client.login(username=user.username, password='test') setattr(api_client, 'user', user) @@ -113,6 +150,10 @@ def superuser_api_client(db, factories, api_client): @pytest.fixture def superuser_client(db, factories, client): + """ + Return a logged-in, non-API client with an authenticated ``User`` + stored in the ``user`` attribute + """ user = factories['users.SuperUser']() assert client.login(username=user.username, password='test') setattr(client, 'user', user) @@ -122,11 +163,17 @@ def superuser_client(db, factories, client): @pytest.fixture def api_request(): + """ + Returns a dummy API request object you can pass to API views + """ return APIRequestFactory() @pytest.fixture def fake_request(): + """ + Returns a dummy, non-API request object you can pass to regular views + """ return client.RequestFactory() @@ -140,16 +187,6 @@ def activity_registry(): record.registry[key] = value -@pytest.fixture -def activity_registry(): - r = record.registry - state = list(record.registry.items()) - yield record.registry - record.registry.clear() - for key, value in state: - record.registry[key] = value - - @pytest.fixture def activity_muted(activity_registry, mocker): yield mocker.patch.object(record, 'send') @@ -157,6 +194,9 @@ def activity_muted(activity_registry, mocker): @pytest.fixture(autouse=True) def media_root(settings): + """ + Sets settings.MEDIA_ROOT to a temporary path and returns this path + """ tmp_dir = tempfile.mkdtemp() settings.MEDIA_ROOT = tmp_dir yield settings.MEDIA_ROOT @@ -165,12 +205,19 @@ def media_root(settings): @pytest.fixture def r_mock(): + """ + Returns a requests_mock.mock() object you can use to mock HTTP calls made + using python-requests + """ with requests_mock.mock() as m: yield m @pytest.fixture def authenticated_actor(factories, mocker): + """ + Returns an authenticated ActivityPub actor + """ actor = factories['federation.Actor']() mocker.patch( 'funkwhale_api.federation.authentication.SignatureAuthentication.authenticate_actor', diff --git a/api/tests/favorites/test_favorites.py b/api/tests/favorites/test_favorites.py index f4a045af8..591fe7c9c 100644 --- a/api/tests/favorites/test_favorites.py +++ b/api/tests/favorites/test_favorites.py @@ -99,8 +99,8 @@ def test_user_can_remove_favorite_via_api_using_track_id( @pytest.mark.parametrize('url,method', [ ('api:v1:favorites:tracks-list', 'get'), ]) -def test_url_require_auth(url, method, db, settings, client): - settings.API_AUTHENTICATION_REQUIRED = True +def test_url_require_auth(url, method, db, preferences, client): + preferences['common__api_authentication_required'] = True url = reverse(url) response = getattr(client, method)(url) assert response.status_code == 401 diff --git a/api/tests/federation/test_actors.py b/api/tests/federation/test_actors.py index 7281147a1..6f73a9b9b 100644 --- a/api/tests/federation/test_actors.py +++ b/api/tests/federation/test_actors.py @@ -29,6 +29,42 @@ def test_actor_fetching(r_mock): assert r == payload +def test_get_actor(factories, r_mock): + actor = factories['federation.Actor'].build() + payload = serializers.ActorSerializer(actor).data + r_mock.get(actor.url, json=payload) + new_actor = actors.get_actor(actor.url) + + assert new_actor.pk is not None + assert serializers.ActorSerializer(new_actor).data == payload + + +def test_get_actor_use_existing(factories, preferences, mocker): + preferences['federation__actor_fetch_delay'] = 60 + actor = factories['federation.Actor']() + get_data = mocker.patch('funkwhale_api.federation.actors.get_actor_data') + new_actor = actors.get_actor(actor.url) + + assert new_actor == actor + get_data.assert_not_called() + + +def test_get_actor_refresh(factories, preferences, mocker): + preferences['federation__actor_fetch_delay'] = 0 + actor = factories['federation.Actor']() + payload = serializers.ActorSerializer(actor).data + # actor changed their username in the meantime + payload['preferredUsername'] = 'New me' + get_data = mocker.patch( + 'funkwhale_api.federation.actors.get_actor_data', + return_value=payload) + new_actor = actors.get_actor(actor.url) + + assert new_actor == actor + assert new_actor.last_fetch_date > actor.last_fetch_date + assert new_actor.preferred_username == 'New me' + + def test_get_library(db, settings, mocker): get_key_pair = mocker.patch( 'funkwhale_api.federation.keys.get_key_pair', @@ -238,9 +274,9 @@ def test_actor_is_system( @pytest.mark.parametrize('value', [False, True]) -def test_library_actor_manually_approves_based_on_setting( - value, settings): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = value +def test_library_actor_manually_approves_based_on_preference( + value, preferences): + preferences['federation__music_needs_approval'] = value library_conf = actors.SYSTEM_ACTORS['library'] assert library_conf.manually_approves_followers is value @@ -320,8 +356,8 @@ def test_test_actor_handles_undo_follow( def test_library_actor_handles_follow_manual_approval( - settings, mocker, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + preferences, mocker, factories): + preferences['federation__music_needs_approval'] = True actor = factories['federation.Actor']() now = timezone.now() mocker.patch('django.utils.timezone.now', return_value=now) @@ -341,8 +377,8 @@ def test_library_actor_handles_follow_manual_approval( def test_library_actor_handles_follow_auto_approval( - settings, mocker, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False + preferences, mocker, factories): + preferences['federation__music_needs_approval'] = False actor = factories['federation.Actor']() accept_follow = mocker.patch( 'funkwhale_api.federation.activity.accept_follow') diff --git a/api/tests/federation/test_permissions.py b/api/tests/federation/test_permissions.py index 9b8683210..a87f26f1b 100644 --- a/api/tests/federation/test_permissions.py +++ b/api/tests/federation/test_permissions.py @@ -5,8 +5,8 @@ from funkwhale_api.federation import permissions def test_library_follower( - factories, api_request, anonymous_user, settings): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + factories, api_request, anonymous_user, preferences): + preferences['federation__music_needs_approval'] = True view = APIView.as_view() permission = permissions.LibraryFollower() request = api_request.get('/') @@ -17,8 +17,8 @@ def test_library_follower( def test_library_follower_actor_non_follower( - factories, api_request, anonymous_user, settings): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + factories, api_request, anonymous_user, preferences): + preferences['federation__music_needs_approval'] = True actor = factories['federation.Actor']() view = APIView.as_view() permission = permissions.LibraryFollower() @@ -31,8 +31,8 @@ def test_library_follower_actor_non_follower( def test_library_follower_actor_follower_not_approved( - factories, api_request, anonymous_user, settings): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + factories, api_request, anonymous_user, preferences): + preferences['federation__music_needs_approval'] = True library = actors.SYSTEM_ACTORS['library'].get_actor_instance() follow = factories['federation.Follow'](target=library, approved=False) view = APIView.as_view() @@ -46,8 +46,8 @@ def test_library_follower_actor_follower_not_approved( def test_library_follower_actor_follower( - factories, api_request, anonymous_user, settings): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + factories, api_request, anonymous_user, preferences): + preferences['federation__music_needs_approval'] = True library = actors.SYSTEM_ACTORS['library'].get_actor_instance() follow = factories['federation.Follow'](target=library, approved=True) view = APIView.as_view() diff --git a/api/tests/federation/test_tasks.py b/api/tests/federation/test_tasks.py index 506fbc1fe..3517e8feb 100644 --- a/api/tests/federation/test_tasks.py +++ b/api/tests/federation/test_tasks.py @@ -1,4 +1,7 @@ import datetime +import os +import pathlib +import pytest from django.core.paginator import Paginator from django.utils import timezone @@ -117,17 +120,16 @@ def test_clean_federation_music_cache_if_no_listen(preferences, factories): lt1 = factories['federation.LibraryTrack'](with_audio_file=True) lt2 = factories['federation.LibraryTrack'](with_audio_file=True) lt3 = factories['federation.LibraryTrack'](with_audio_file=True) - tf1 = factories['music.TrackFile'](library_track=lt1) - tf2 = factories['music.TrackFile'](library_track=lt2) - tf3 = factories['music.TrackFile'](library_track=lt3) - - # we listen to the first one, and the second one (but weeks ago) - listening1 = factories['history.Listening']( - track=tf1.track, - creation_date=timezone.now()) - listening2 = factories['history.Listening']( - track=tf2.track, - creation_date=timezone.now() - datetime.timedelta(minutes=61)) + tf1 = factories['music.TrackFile']( + accessed_date=timezone.now(), library_track=lt1) + tf2 = factories['music.TrackFile']( + accessed_date=timezone.now()-datetime.timedelta(minutes=61), + library_track=lt2) + tf3 = factories['music.TrackFile']( + accessed_date=None, library_track=lt3) + path1 = lt1.audio_file.path + path2 = lt2.audio_file.path + path3 = lt3.audio_file.path tasks.clean_music_cache() @@ -138,3 +140,32 @@ def test_clean_federation_music_cache_if_no_listen(preferences, factories): assert bool(lt1.audio_file) is True assert bool(lt2.audio_file) is False assert bool(lt3.audio_file) is False + assert os.path.exists(path1) is True + assert os.path.exists(path2) is False + assert os.path.exists(path3) is False + + +def test_clean_federation_music_cache_orphaned( + settings, preferences, factories): + preferences['federation__music_cache_duration'] = 60 + path = os.path.join(settings.MEDIA_ROOT, 'federation_cache') + keep_path = os.path.join(os.path.join(path, '1a', 'b2'), 'keep.ogg') + remove_path = os.path.join(os.path.join(path, 'c3', 'd4'), 'remove.ogg') + os.makedirs(os.path.dirname(keep_path), exist_ok=True) + os.makedirs(os.path.dirname(remove_path), exist_ok=True) + pathlib.Path(keep_path).touch() + pathlib.Path(remove_path).touch() + lt = factories['federation.LibraryTrack']( + with_audio_file=True, + audio_file__path=keep_path) + tf = factories['music.TrackFile']( + library_track=lt, + accessed_date=timezone.now()) + + tasks.clean_music_cache() + + lt.refresh_from_db() + + assert bool(lt.audio_file) is True + assert os.path.exists(lt.audio_file.path) is True + assert os.path.exists(remove_path) is False diff --git a/api/tests/federation/test_views.py b/api/tests/federation/test_views.py index ae94bcdc0..09ecfc8ff 100644 --- a/api/tests/federation/test_views.py +++ b/api/tests/federation/test_views.py @@ -13,7 +13,7 @@ from funkwhale_api.federation import webfinger @pytest.mark.parametrize('system_actor', actors.SYSTEM_ACTORS.keys()) -def test_instance_actors(system_actor, db, settings, api_client): +def test_instance_actors(system_actor, db, api_client): actor = actors.SYSTEM_ACTORS[system_actor].get_actor_instance() url = reverse( 'federation:instance-actors-detail', @@ -34,8 +34,8 @@ def test_instance_actors(system_actor, db, settings, api_client): ('well-known-webfinger', {}), ]) def test_instance_endpoints_405_if_federation_disabled( - authenticated_actor, db, settings, api_client, route, kwargs): - settings.FEDERATION_ENABLED = False + authenticated_actor, db, preferences, api_client, route, kwargs): + preferences['federation__enabled'] = False url = reverse('federation:{}'.format(route), kwargs=kwargs) response = api_client.get(url) @@ -71,8 +71,8 @@ def test_wellknown_webfinger_system( def test_audio_file_list_requires_authenticated_actor( - db, settings, api_client): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = True + db, preferences, api_client): + preferences['federation__music_needs_approval'] = True url = reverse('federation:music:files-list') response = api_client.get(url) @@ -80,9 +80,9 @@ def test_audio_file_list_requires_authenticated_actor( def test_audio_file_list_actor_no_page( - db, settings, api_client, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False - settings.FEDERATION_COLLECTION_PAGE_SIZE = 2 + db, preferences, api_client, factories): + preferences['federation__music_needs_approval'] = False + preferences['federation__collection_page_size'] = 2 library = actors.SYSTEM_ACTORS['library'].get_actor_instance() tfs = factories['music.TrackFile'].create_batch(size=5) conf = { @@ -101,9 +101,9 @@ def test_audio_file_list_actor_no_page( def test_audio_file_list_actor_page( - db, settings, api_client, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False - settings.FEDERATION_COLLECTION_PAGE_SIZE = 2 + db, preferences, api_client, factories): + preferences['federation__music_needs_approval'] = False + preferences['federation__collection_page_size'] = 2 library = actors.SYSTEM_ACTORS['library'].get_actor_instance() tfs = factories['music.TrackFile'].create_batch(size=5) conf = { @@ -121,8 +121,8 @@ def test_audio_file_list_actor_page( def test_audio_file_list_actor_page_exclude_federated_files( - db, settings, api_client, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False + db, preferences, api_client, factories): + preferences['federation__music_needs_approval'] = False library = actors.SYSTEM_ACTORS['library'].get_actor_instance() tfs = factories['music.TrackFile'].create_batch(size=5, federation=True) @@ -134,8 +134,8 @@ def test_audio_file_list_actor_page_exclude_federated_files( def test_audio_file_list_actor_page_error( - db, settings, api_client, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False + db, preferences, api_client, factories): + preferences['federation__music_needs_approval'] = False url = reverse('federation:music:files-list') response = api_client.get(url, data={'page': 'nope'}) @@ -143,15 +143,15 @@ def test_audio_file_list_actor_page_error( def test_audio_file_list_actor_page_error_too_far( - db, settings, api_client, factories): - settings.FEDERATION_MUSIC_NEEDS_APPROVAL = False + db, preferences, api_client, factories): + preferences['federation__music_needs_approval'] = False url = reverse('federation:music:files-list') response = api_client.get(url, data={'page': 5000}) assert response.status_code == 404 -def test_library_actor_includes_library_link(db, settings, api_client): +def test_library_actor_includes_library_link(db, preferences, api_client): actor = actors.SYSTEM_ACTORS['library'].get_actor_instance() url = reverse( 'federation:instance-actors-detail', diff --git a/api/tests/history/test_history.py b/api/tests/history/test_history.py index ec8689e96..202725596 100644 --- a/api/tests/history/test_history.py +++ b/api/tests/history/test_history.py @@ -14,20 +14,6 @@ def test_can_create_listening(factories): l = models.Listening.objects.create(user=user, track=track) -def test_anonymous_user_can_create_listening_via_api(client, factories, settings): - settings.API_AUTHENTICATION_REQUIRED = False - track = factories['music.Track']() - url = reverse('api:v1:history:listenings-list') - response = client.post(url, { - 'track': track.pk, - }) - - listening = models.Listening.objects.latest('id') - - assert listening.track == track - assert listening.session_key == client.session.session_key - - def test_logged_in_user_can_create_listening_via_api( logged_in_client, factories, activity_muted): track = factories['music.Track']() diff --git a/api/tests/music/conftest.py b/api/tests/music/conftest.py index 1d0fa4e38..4eea8effe 100644 --- a/api/tests/music/conftest.py +++ b/api/tests/music/conftest.py @@ -508,21 +508,25 @@ _works['get']['chop_suey'] = {'work': {'id': 'e2ecabc4-1b9d-30b2-8f30-3596ec423d @pytest.fixture() def artists(): + """Artists as they would be returned by the Musicbrainz API""" return _artists @pytest.fixture() def albums(): + """Releases as they would be returned by the Musicbrainz API""" return _albums @pytest.fixture() def tracks(): + """Recordings as they would be returned by the Musicbrainz API""" return _tracks @pytest.fixture() def works(): + """Works as they would be returned by the Musicbrainz API""" return _works @@ -563,4 +567,7 @@ def lyricswiki_content(): @pytest.fixture() def binary_cover(): + """ + Return an album cover image in form of a binary string + """ return b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x02\x01\x00H\x00H\x00\x00\xff\xed\x08\xaePhotoshop 3.0\x008BIM\x03\xe9\x00\x00\x00\x00\x00x\x00\x03\x00\x00\x00H\x00H\x00\x00\x00\x00\x02\xd8\x02(\xff\xe1\xff\xe2\x02\xf9\x02F\x03G\x05(\x03\xfc\x00\x02\x00\x00\x00H\x00H\x00\x00\x00\x00\x02\xd8\x02(\x00\x01\x00\x00\x00d\x00\x00\x00\x01\x00\x03\x03\x03\x00\x00\x00\x01\'\x0f\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00`\x08\x00\x19\x01\x90\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x008BIM\x03\xed\x00\x00\x00\x00\x00\x10\x00H\x00\x00\x00\x01\x00\x01\x00H\x00\x00\x00\x01\x00\x018BIM\x03\xf3\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x008BIM\x04\n\x00\x00\x00\x00\x00\x01\x00\x008BIM\'\x10\x00\x00\x00\x00\x00\n\x00\x01\x00\x00\x00\x00\x00\x00\x00\x028BIM\x03\xf5\x00\x00\x00\x00\x00H\x00/ff\x00\x01\x00lff\x00\x06\x00\x00\x00\x00\x00\x01\x00/ff\x00\x01\x00\xa1\x99\x9a\x00\x06\x00\x00\x00\x00\x00\x01\x002\x00\x00\x00\x01\x00Z\x00\x00\x00\x06\x00\x00\x00\x00\x00\x01\x005\x00\x00\x00\x01\x00-\x00\x00\x00\x06\x00\x00\x00\x00\x00\x018BIM\x03\xf8\x00\x00\x00\x00\x00p\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x03\xe8\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x03\xe8\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x03\xe8\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x03\xe8\x00\x008BIM\x04\x00\x00\x00\x00\x00\x00\x02\x00\x018BIM\x04\x02\x00\x00\x00\x00\x00\x04\x00\x00\x00\x008BIM\x04\x08\x00\x00\x00\x00\x00\x10\x00\x00\x00\x01\x00\x00\x02@\x00\x00\x02@\x00\x00\x00\x008BIM\x04\t\x00\x00\x00\x00\x06\x9b\x00\x00\x00\x01\x00\x00\x00\x80\x00\x00\x00\x80\x00\x00\x01\x80\x00\x00\xc0\x00\x00\x00\x06\x7f\x00\x18\x00\x01\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x02\x01\x00H\x00H\x00\x00\xff\xfe\x00\'File written by Adobe Photoshop\xa8 4.0\x00\xff\xee\x00\x0eAdobe\x00d\x80\x00\x00\x00\x01\xff\xdb\x00\x84\x00\x0c\x08\x08\x08\t\x08\x0c\t\t\x0c\x11\x0b\n\x0b\x11\x15\x0f\x0c\x0c\x0f\x15\x18\x13\x13\x15\x13\x13\x18\x11\x0c\x0c\x0c\x0c\x0c\x0c\x11\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x01\r\x0b\x0b\r\x0e\r\x10\x0e\x0e\x10\x14\x0e\x0e\x0e\x14\x14\x0e\x0e\x0e\x0e\x14\x11\x0c\x0c\x0c\x0c\x0c\x11\x11\x0c\x0c\x0c\x0c\x0c\x0c\x11\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\xff\xc0\x00\x11\x08\x00\x80\x00\x80\x03\x01"\x00\x02\x11\x01\x03\x11\x01\xff\xdd\x00\x04\x00\x08\xff\xc4\x01?\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x03\x00\x01\x02\x04\x05\x06\x07\x08\t\n\x0b\x01\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x10\x00\x01\x04\x01\x03\x02\x04\x02\x05\x07\x06\x08\x05\x03\x0c3\x01\x00\x02\x11\x03\x04!\x121\x05AQa\x13"q\x812\x06\x14\x91\xa1\xb1B#$\x15R\xc1b34r\x82\xd1C\x07%\x92S\xf0\xe1\xf1cs5\x16\xa2\xb2\x83&D\x93TdE\xc2\xa3t6\x17\xd2U\xe2e\xf2\xb3\x84\xc3\xd3u\xe3\xf3F\'\x94\xa4\x85\xb4\x95\xc4\xd4\xe4\xf4\xa5\xb5\xc5\xd5\xe5\xf5Vfv\x86\x96\xa6\xb6\xc6\xd6\xe6\xf67GWgw\x87\x97\xa7\xb7\xc7\xd7\xe7\xf7\x11\x00\x02\x02\x01\x02\x04\x04\x03\x04\x05\x06\x07\x07\x06\x055\x01\x00\x02\x11\x03!1\x12\x04AQaq"\x13\x052\x81\x91\x14\xa1\xb1B#\xc1R\xd1\xf03$b\xe1r\x82\x92CS\x15cs4\xf1%\x06\x16\xa2\xb2\x83\x07&5\xc2\xd2D\x93T\xa3\x17dEU6te\xe2\xf2\xb3\x84\xc3\xd3u\xe3\xf3F\x94\xa4\x85\xb4\x95\xc4\xd4\xe4\xf4\xa5\xb5\xc5\xd5\xe5\xf5Vfv\x86\x96\xa6\xb6\xc6\xd6\xe6\xf6\'7GWgw\x87\x97\xa7\xb7\xc7\xff\xda\x00\x0c\x03\x01\x00\x02\x11\x03\x11\x00?\x00\xf5T\x92I%)$\x92IJI$\x92R\x92I$\x94\xa4\x92I%)$\x92IJI$\x92R\x92I$\x94\xff\x00\xff\xd0\xf5T\x92I%)$\x92IJI%\xe7\xff\x00Z\x7f\xc6\xbf\xfc\xde\xeb\xb9]\x1f\xf6_\xda~\xcd\xe9\xfe\x9b\xed\x1e\x9e\xefR\xba\xef\xfeo\xec\xf6\xed\xdb\xea\xec\xfeq%>\x80\x92\xf2\xaf\xfc}?\xf3I\xff\x00\xb3_\xfb\xe8\x97\xfe>\x9f\xf9\xa4\xff\x00\xd9\xaf\xfd\xf4IO\xaa\xa4\xbc\xab\xff\x00\x1fO\xfc\xd2\x7f\xec\xd7\xfe\xfa%\xff\x00\x8f\xa7\xfei?\xf6k\xff\x00}\x12S\xea\xa9.+\xeaW\xf8\xc8\xff\x00\x9d}V\xde\x9d\xfb;\xec~\x96;\xb2=O[\xd5\x9d\xaf\xaa\xad\x9b=\n\x7f\xd3}-\xeb\xb5IJI$\x92R\x92I$\x94\xff\x00\xff\xd1\xf5T\x92I%)$\x97\x9f\xff\x00\x8d\x7f\xad=w\xea\xf7\xec\xbf\xd8\xf9_f\xfbO\xda=o\xd1\xd7f\xefO\xec\xfe\x9f\xf3\xf5\xdb\xb7o\xabg\xd0IO\xa0/\x9f\xff\x00\xc6\x97\xfe.\xfa\x9f\xfdc\xff\x00m\xf1\xd2\xff\x00\xc7K\xeb\xdf\xfeY\xff\x00\xe0\x18\xff\x00\xfb\xce\xb9\xfe\xa9\xd53\xfa\xbe}\xbdG\xa8\xdb\xeb\xe5\xdf\xb7\xd4\xb3kY;\x1a\xda\x99\xec\xa9\xac\xaf\xf9\xb63\xf3\x12SU$\x92IJI$\x92S\xdf\xff\x00\x89O\xfcUe\x7f\xe1\x0b?\xf3\xf6*\xf6\xb5\xf3/D\xeb\xfd[\xa0\xe5?3\xa4\xdf\xf6l\x8b+59\xfb\x18\xf9a-\xb1\xcd\xdb{-g\xd3\xa9\x8bk\xff\x00\x1d/\xaf\x7f\xf9g\xff\x00\x80c\xff\x00\xef:J~\x80Iq\xff\x00\xe2\xbf\xaf\xf5n\xbd\xd023:\xb5\xff\x00i\xc8\xaf-\xf55\xfb\x18\xc8`\xae\x8b\x1a\xdd\xb42\xa6};^\xbb\x04\x94\xa4\x92I%?\xff\xd2\xf5T\x92I%)yW\xf8\xf4\xff\x00\xbcO\xfd\n\xff\x00\xddE\xea\xab\xca\xbf\xc7\xa7\xfd\xe2\x7f\xe8W\xfe\xea$\xa7\xca\x92I$\x94\xa4\x92I%)$\x92IJI$\x92S\xed_\xe2S\xff\x00\x12\xb9_\xf8~\xcf\xfc\xf3\x8a\xbd\x01y\xff\x00\xf8\x94\xff\x00\xc4\xaeW\xfe\x1f\xb3\xff\x00<\xe2\xaf@IJI$\x92S\xff\xd3\xf5T\x92I%)yW\xf8\xf4\xff\x00\xbcO\xfd\n\xff\x00\xddE\xea\xab\xca\xbf\xc7\xa7\xfd\xe2\x7f\xe8W\xfe\xea$\xa7\xca\x92I$\x94\xa4\x92I%)$\x92IJI$\x92S\xed_\xe2S\xff\x00\x12\xb9_\xf8~\xcf\xfc\xf3\x8a\xbd\x01y\xff\x00\xf8\x94\xff\x00\xc4\xaeW\xfe\x1f\xb3\xff\x00<\xe2\xaf@IJI$\x92S\xff\xd4\xf5T\x92I%)q_\xe3#\xeaWU\xfa\xd7\xfb;\xf6u\xb8\xf5}\x8f\xd6\xf5>\xd0\xe7\xb6}_Cf\xcfJ\xab\xbf\xd0\xbfr\xedRIO\x8a\x7f\xe3)\xf5\xab\xfe\xe5`\x7f\xdb\x97\x7f\xef*\xe4:\xff\x00D\xca\xe8=Z\xfe\x93\x98\xfa\xec\xc8\xc6\xd9\xbd\xd5\x12Xw\xb1\x97\xb7k\xacmO\xfa\x16\xfe\xe2\xfai|\xff\x00\xfe4\xbf\xf1w\xd4\xff\x00\xeb\x1f\xfbo\x8e\x92\x9eU$\x92IJI$\x92S\xb1\xf5_\xea\xbfP\xfa\xd1\xd4,\xc0\xc0\xb2\x9a\xad\xaa\x93{\x9dys[\xb5\xae\xae\xa2\x01\xaa\xbb\x9d\xbfu\xcd\xfc\xd5\xd3\xff\x00\xe3)\xf5\xab\xfe\xe5`\x7f\xdb\x97\x7f\xef*_\xe2S\xff\x00\x15Y_\xf8B\xcf\xfc\xfd\x8a\xbd\xad%<\xbf\xf8\xbc\xfa\xaf\xd4>\xab\xf4[\xb03\xec\xa6\xdbm\xc9u\xedu\x05\xcen\xd7WM@\x13mt\xbb~\xea]\xf9\xab\xa8I$\x94\xa4\x92I%?\xff\xd5\xf5T\x92I%)$\x92IJ\\\x7f_\xff\x00\x15\xfd\x03\xafuk\xfa\xb6fF]y\x19;7\xb6\xa7\xd6\x1861\x947kl\xa2\xd7\xfd\n\xbf}v\t$\xa7\xcf\xff\x00\xf1\x94\xfa\xab\xff\x00r\xb3\xff\x00\xed\xca\x7f\xf7\x95/\xfce>\xaa\xff\x00\xdc\xac\xff\x00\xfbr\x9f\xfd\xe5^\x80\x92J|\xff\x00\xff\x00\x19O\xaa\xbf\xf7+?\xfe\xdc\xa7\xff\x00yR\xff\x00\xc6S\xea\xaf\xfd\xca\xcf\xff\x00\xb7)\xff\x00\xdeU\xe8\t$\xa7\x97\xfa\xaf\xfe/:/\xd5~\xa1f~\x05\xd96\xdbm&\x876\xf7V\xe6\xeds\xab\xb4\x90*\xa6\x97o\xddK\x7f9u\t$\x92\x94\x92I$\xa5$\x92I)\xff\xd6\xf5T\x92I%)$\x92IJI$\x92R\x92I$\x94\xa4\x92I%)$\x92IJI$\x92R\x92I$\x94\xff\x00\xff\xd9\x008BIM\x04\x06\x00\x00\x00\x00\x00\x07\x00\x03\x00\x00\x00\x01\x01\x00\xff\xfe\x00\'File written by Adobe Photoshop\xa8 4.0\x00\xff\xee\x00\x0eAdobe\x00d\x00\x00\x00\x00\x01\xff\xdb\x00\x84\x00\n\x07\x07\x07\x08\x07\n\x08\x08\n\x0f\n\x08\n\x0f\x12\r\n\n\r\x12\x14\x10\x10\x12\x10\x10\x14\x11\x0c\x0c\x0c\x0c\x0c\x0c\x11\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x01\x0b\x0c\x0c\x15\x13\x15"\x18\x18"\x14\x0e\x0e\x0e\x14\x14\x0e\x0e\x0e\x0e\x14\x11\x0c\x0c\x0c\x0c\x0c\x11\x11\x0c\x0c\x0c\x0c\x0c\x0c\x11\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\x0c\xff\xc0\x00\x11\x08\x00\t\x00\t\x03\x01\x11\x00\x02\x11\x01\x03\x11\x01\xff\xdd\x00\x04\x00\x02\xff\xc4\x01\xa2\x00\x00\x00\x07\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x05\x03\x02\x06\x01\x00\x07\x08\t\n\x0b\x01\x00\x02\x02\x03\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x10\x00\x02\x01\x03\x03\x02\x04\x02\x06\x07\x03\x04\x02\x06\x02s\x01\x02\x03\x11\x04\x00\x05!\x121AQ\x06\x13a"q\x81\x142\x91\xa1\x07\x15\xb1B#\xc1R\xd1\xe13\x16b\xf0$r\x82\xf1%C4S\x92\xa2\xb2cs\xc25D\'\x93\xa3\xb36\x17Tdt\xc3\xd2\xe2\x08&\x83\t\n\x18\x19\x84\x94EF\xa4\xb4V\xd3U(\x1a\xf2\xe3\xf3\xc4\xd4\xe4\xf4eu\x85\x95\xa5\xb5\xc5\xd5\xe5\xf5fv\x86\x96\xa6\xb6\xc6\xd6\xe6\xf67GWgw\x87\x97\xa7\xb7\xc7\xd7\xe7\xf78HXhx\x88\x98\xa8\xb8\xc8\xd8\xe8\xf8)9IYiy\x89\x99\xa9\xb9\xc9\xd9\xe9\xf9*:JZjz\x8a\x9a\xaa\xba\xca\xda\xea\xfa\x11\x00\x02\x02\x01\x02\x03\x05\x05\x04\x05\x06\x04\x08\x03\x03m\x01\x00\x02\x11\x03\x04!\x121A\x05Q\x13a"\x06q\x81\x912\xa1\xb1\xf0\x14\xc1\xd1\xe1#B\x15Rbr\xf13$4C\x82\x16\x92S%\xa2c\xb2\xc2\x07s\xd25\xe2D\x83\x17T\x93\x08\t\n\x18\x19&6E\x1a\'dtU7\xf2\xa3\xb3\xc3()\xd3\xe3\xf3\x84\x94\xa4\xb4\xc4\xd4\xe4\xf4eu\x85\x95\xa5\xb5\xc5\xd5\xe5\xf5FVfv\x86\x96\xa6\xb6\xc6\xd6\xe6\xf6GWgw\x87\x97\xa7\xb7\xc7\xd7\xe7\xf78HXhx\x88\x98\xa8\xb8\xc8\xd8\xe8\xf89IYiy\x89\x99\xa9\xb9\xc9\xd9\xe9\xf9*:JZjz\x8a\x9a\xaa\xba\xca\xda\xea\xfa\xff\xda\x00\x0c\x03\x01\x00\x02\x11\x03\x11\x00?\x00\x91\xea\xfa\xbf\xe6D_\x99\x16\x96\x16\x16\x8c\xdeWf\x84;\x88U\xa1hY\x7f\xd3\'\x9e\xf3\xedCq\x0bz\xfe\x94^\xbc?\xdc\xdb\xff\x00\xa3\xcd\xeb\x7f\xa4\xaa\xf4 now def test_can_create_import_from_federation_tracks( @@ -196,3 +222,64 @@ def test_import_job_stats_filter(factories, superuser_api_client): } assert response.status_code == 200 assert response.data == expected + + +def test_import_job_run_via_api(factories, superuser_api_client, mocker): + run = mocker.patch('funkwhale_api.music.tasks.import_job_run.delay') + job1 = factories['music.ImportJob'](status='errored') + job2 = factories['music.ImportJob'](status='pending') + + url = reverse('api:v1:import-jobs-run') + response = superuser_api_client.post(url, {'jobs': [job2.pk, job1.pk]}) + + job1.refresh_from_db() + job2.refresh_from_db() + assert response.status_code == 200 + assert response.data == {'jobs': [job1.pk, job2.pk]} + assert job1.status == 'pending' + assert job2.status == 'pending' + + run.assert_any_call(import_job_id=job1.pk) + run.assert_any_call(import_job_id=job2.pk) + + +def test_import_batch_run_via_api(factories, superuser_api_client, mocker): + run = mocker.patch('funkwhale_api.music.tasks.import_job_run.delay') + + batch = factories['music.ImportBatch']() + job1 = factories['music.ImportJob'](batch=batch, status='errored') + job2 = factories['music.ImportJob'](batch=batch, status='pending') + + url = reverse('api:v1:import-jobs-run') + response = superuser_api_client.post(url, {'batches': [batch.pk]}) + + job1.refresh_from_db() + job2.refresh_from_db() + assert response.status_code == 200 + assert job1.status == 'pending' + assert job2.status == 'pending' + + run.assert_any_call(import_job_id=job1.pk) + run.assert_any_call(import_job_id=job2.pk) + + +def test_import_batch_and_job_run_via_api( + factories, superuser_api_client, mocker): + run = mocker.patch('funkwhale_api.music.tasks.import_job_run.delay') + + batch = factories['music.ImportBatch']() + job1 = factories['music.ImportJob'](batch=batch, status='errored') + job2 = factories['music.ImportJob'](status='pending') + + url = reverse('api:v1:import-jobs-run') + response = superuser_api_client.post( + url, {'batches': [batch.pk], 'jobs': [job2.pk]}) + + job1.refresh_from_db() + job2.refresh_from_db() + assert response.status_code == 200 + assert job1.status == 'pending' + assert job2.status == 'pending' + + run.assert_any_call(import_job_id=job1.pk) + run.assert_any_call(import_job_id=job2.pk) diff --git a/api/tests/playlists/test_models.py b/api/tests/playlists/test_models.py index c9def4dab..fe5dd40a8 100644 --- a/api/tests/playlists/test_models.py +++ b/api/tests/playlists/test_models.py @@ -116,8 +116,8 @@ def test_can_insert_many(factories): assert plt.playlist == playlist -def test_insert_many_honor_max_tracks(factories, settings): - settings.PLAYLISTS_MAX_TRACKS = 4 +def test_insert_many_honor_max_tracks(preferences, factories): + preferences['playlists__max_tracks'] = 4 playlist = factories['playlists.Playlist']() plts = factories['playlists.PlaylistTrack'].create_batch( size=2, playlist=playlist) diff --git a/api/tests/playlists/test_serializers.py b/api/tests/playlists/test_serializers.py index 8e30919e6..908c1c796 100644 --- a/api/tests/playlists/test_serializers.py +++ b/api/tests/playlists/test_serializers.py @@ -2,8 +2,8 @@ from funkwhale_api.playlists import models from funkwhale_api.playlists import serializers -def test_cannot_max_500_tracks_per_playlist(factories, settings): - settings.PLAYLISTS_MAX_TRACKS = 2 +def test_cannot_max_500_tracks_per_playlist(factories, preferences): + preferences['playlists__max_tracks'] = 2 playlist = factories['playlists.Playlist']() plts = factories['playlists.PlaylistTrack'].create_batch( size=2, playlist=playlist) diff --git a/api/tests/playlists/test_views.py b/api/tests/playlists/test_views.py index f0fb6d0fd..44d060821 100644 --- a/api/tests/playlists/test_views.py +++ b/api/tests/playlists/test_views.py @@ -107,8 +107,8 @@ def test_deleting_plt_updates_indexes( @pytest.mark.parametrize('level', ['instance', 'me', 'followers']) def test_playlist_privacy_respected_in_list_anon( - settings, level, factories, api_client): - settings.API_AUTHENTICATION_REQUIRED = False + preferences, level, factories, api_client): + preferences['common__api_authentication_required'] = False factories['playlists.Playlist'](privacy_level=level) url = reverse('api:v1:playlists-list') response = api_client.get(url) @@ -137,8 +137,8 @@ def test_only_owner_can_edit_playlist_track( @pytest.mark.parametrize('level', ['instance', 'me', 'followers']) def test_playlist_track_privacy_respected_in_list_anon( - level, factories, api_client, settings): - settings.API_AUTHENTICATION_REQUIRED = False + level, factories, api_client, preferences): + preferences['common__api_authentication_required'] = False factories['playlists.PlaylistTrack'](playlist__privacy_level=level) url = reverse('api:v1:playlist-tracks-list') response = api_client.get(url) diff --git a/api/tests/radios/test_radios.py b/api/tests/radios/test_radios.py index c8038a4db..b166b648c 100644 --- a/api/tests/radios/test_radios.py +++ b/api/tests/radios/test_radios.py @@ -151,20 +151,6 @@ def test_can_start_radio_for_logged_in_user(logged_in_client): assert session.user == logged_in_client.user -def test_can_start_radio_for_anonymous_user(api_client, db, settings): - settings.API_AUTHENTICATION_REQUIRED = False - url = reverse('api:v1:radios:sessions-list') - response = api_client.post(url, {'radio_type': 'random'}) - - assert response.status_code == 201 - - session = models.RadioSession.objects.latest('id') - - assert session.radio_type == 'random' - assert session.user is None - assert session.session_key == api_client.session.session_key - - def test_can_get_track_for_session_from_api(factories, logged_in_client): files = factories['music.TrackFile'].create_batch(1) tracks = [f.track for f in files] @@ -227,25 +213,25 @@ def test_can_start_tag_radio(factories): radio = radios.TagRadio() session = radio.start_session(user, related_object=tag) - assert session.radio_type =='tag' + assert session.radio_type == 'tag' for i in range(5): assert radio.pick() in good_tracks -def test_can_start_artist_radio_from_api(api_client, settings, factories): - settings.API_AUTHENTICATION_REQUIRED = False +def test_can_start_artist_radio_from_api( + logged_in_api_client, preferences, factories): artist = factories['music.Artist']() url = reverse('api:v1:radios:sessions-list') - response = api_client.post( + response = logged_in_api_client.post( url, {'radio_type': 'artist', 'related_object_id': artist.id}) assert response.status_code == 201 session = models.RadioSession.objects.latest('id') - assert session.radio_type, 'artist' - assert session.related_object, artist + assert session.radio_type == 'artist' + assert session.related_object == artist def test_can_start_less_listened_radio(factories): @@ -257,6 +243,6 @@ def test_can_start_less_listened_radio(factories): good_tracks = [f.track for f in good_files] radio = radios.LessListenedRadio() session = radio.start_session(user) - assert session.related_object == user + for i in range(5): assert radio.pick() in good_tracks diff --git a/api/tests/test_jwt_querystring.py b/api/tests/test_jwt_querystring.py index bd07e1dc3..f18e6b729 100644 --- a/api/tests/test_jwt_querystring.py +++ b/api/tests/test_jwt_querystring.py @@ -5,9 +5,10 @@ jwt_payload_handler = api_settings.JWT_PAYLOAD_HANDLER jwt_encode_handler = api_settings.JWT_ENCODE_HANDLER -def test_can_authenticate_using_token_param_in_url(factories, settings, client): +def test_can_authenticate_using_token_param_in_url( + factories, preferences, client): user = factories['users.User']() - settings.API_AUTHENTICATION_REQUIRED = True + preferences['common__api_authentication_required'] = True url = reverse('api:v1:tracks-list') response = client.get(url) diff --git a/api/tests/test_youtube.py b/api/tests/test_youtube.py index 441179095..7ab6256da 100644 --- a/api/tests/test_youtube.py +++ b/api/tests/test_youtube.py @@ -18,8 +18,8 @@ def test_can_get_search_results_from_youtube(mocker): def test_can_get_search_results_from_funkwhale( - settings, mocker, api_client, db): - settings.API_AUTHENTICATION_REQUIRED = False + preferences, mocker, api_client, db): + preferences['common__api_authentication_required'] = False mocker.patch( 'funkwhale_api.providers.youtube.client._do_search', return_value=api_data.search['8 bit adventure']) @@ -70,8 +70,8 @@ def test_can_send_multiple_queries_at_once(mocker): def test_can_send_multiple_queries_at_once_from_funwkhale( - settings, mocker, db, api_client): - settings.API_AUTHENTICATION_REQUIRED = False + preferences, mocker, db, api_client): + preferences['common__api_authentication_required'] = False mocker.patch( 'funkwhale_api.providers.youtube.client._do_search', return_value=api_data.search['8 bit adventure']) diff --git a/api/tests/users/test_views.py b/api/tests/users/test_views.py index 4be586965..985a78c8a 100644 --- a/api/tests/users/test_views.py +++ b/api/tests/users/test_views.py @@ -136,6 +136,20 @@ def test_changing_password_updates_secret_key(logged_in_client): assert user.password != password +def test_can_request_password_reset( + factories, api_client, mailoutbox): + user = factories['users.User']() + payload = { + 'email': user.email, + } + emails = len(mailoutbox) + url = reverse('rest_password_reset') + + response = api_client.post(url, payload) + assert response.status_code == 200 + assert len(mailoutbox) > emails + + def test_user_can_patch_his_own_settings(logged_in_api_client): user = logged_in_api_client.user payload = { diff --git a/changes/template.rst b/changes/template.rst index f4d94dee8..24f0e87eb 100644 --- a/changes/template.rst +++ b/changes/template.rst @@ -1,3 +1,6 @@ + +Upgrade instructions are available at https://docs.funkwhale.audio/upgrading.html + {% for section, _ in sections.items() %} {% if sections[section] %} {% for category, val in definitions.items() if category in sections[section]%} diff --git a/deploy/apache.conf b/deploy/apache.conf new file mode 100644 index 000000000..8d5a5e1f7 --- /dev/null +++ b/deploy/apache.conf @@ -0,0 +1,122 @@ +# Following variables MUST be modified according to your setup +Define funkwhale-sn funkwhale.yourdomain.com + +# Following variables should be modified according to your setup and if you +# use different configuration than what is described in our installation guide. +Define funkwhale-api http://localhost:5000 +Define MUSIC_DIRECTORY_PATH /srv/funkwhale/data/music +# websockets are not working yet +# Define funkwhale-api-ws ws://localhost:5000 + + +# HTTP request redirected to HTTPS + + ServerName ${funkwhale-sn} + + # Default is to force https + RewriteEngine on + RewriteCond %{SERVER_NAME} =${funkwhale-sn} + RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent] + + + Options None + Require all granted + + + + + + + + ServerName ${funkwhale-sn} + + # Path to ErrorLog and access log + ErrorLog ${APACHE_LOG_DIR}/funkwhale/error.log + CustomLog ${APACHE_LOG_DIR}/funkwhale/access.log combined + + # TLS + # Feel free to use your own configuration for SSL here or simply remove the + # lines and move the configuration to the previous server block if you + # don't want to run funkwhale behind https (this is not recommanded) + # have a look here for let's encrypt configuration: + # https://certbot.eff.org/all-instructions/#debian-9-stretch-nginx + SSLEngine on + SSLProxyEngine On + SSLCertificateFile /etc/letsencrypt/live/${funkwhale-sn}/fullchain.pem + SSLCertificateKeyFile /etc/letsencrypt/live/${funkwhale-sn}/privkey.pem + Include /etc/letsencrypt/options-ssl-apache.conf + + + DocumentRoot /srv/funkwhale/front/dist + + FallbackResource /index.html + + # Configure Proxy settings + # ProxyPreserveHost pass the original Host header to the backend server + ProxyVia On + ProxyPreserveHost On + + RemoteIPHeader X-Forwarded-For + + + # Turning ProxyRequests on and allowing proxying from all may allow + # spammers to use your proxy to send email. + ProxyRequests Off + + + AddDefaultCharset off + Order Allow,Deny + Allow from all + + + # Activating WebSockets (not working) + # ProxyPass "/api/v1/instance/activity" "ws://localhost:5000/api/v1/instance/activity" + + + # similar to nginx 'client_max_body_size 30M;' + LimitRequestBody 31457280 + + ProxyPass ${funkwhale-api}/api + ProxyPassReverse ${funkwhale-api}/api + + + ProxyPass ${funkwhale-api}/federation + ProxyPassReverse ${funkwhale-api}/federation + + + + ProxyPass ${funkwhale-api}/.well-known/webfinger + ProxyPassReverse ${funkwhale-api}/.well-known/webfinger + + + Alias /media /srv/funkwhale/data/media + + Alias /staticfiles /srv/funkwhale/data/static + + # Setting appropriate access levels to serve frontend + + Options FollowSymLinks + AllowOverride None + Require all granted + + + + Options FollowSymLinks + AllowOverride None + Require all granted + + + # XSendFile is serving audio files + # WARNING : permissions on paths specified below overrides previous definition, + # everything under those paths is potentially exposed. + # Following directive may be needed to ensure xsendfile is loaded + #LoadModule xsendfile_module modules/mod_xsendfile.so + + XSendFile On + XSendFilePath /srv/funkwhale/data/media + XSendFilePath ${MUSIC_DIRECTORY_PATH} + SetEnv MOD_X_SENDFILE_ENABLED 1 + + + + diff --git a/deploy/env.prod.sample b/deploy/env.prod.sample index 54f2e1ef0..4b27595af 100644 --- a/deploy/env.prod.sample +++ b/deploy/env.prod.sample @@ -6,6 +6,7 @@ # - DJANGO_SECRET_KEY # - DJANGO_ALLOWED_HOSTS # - FUNKWHALE_URL +# - EMAIL_CONFIG and DEFAULT_FROM_EMAIL if you plan to send emails) # On non-docker setup **only**, you'll also have to tweak/uncomment those variables: # - DATABASE_URL # - CACHE_URL @@ -41,6 +42,19 @@ FUNKWHALE_API_PORT=5000 # your instance FUNKWHALE_URL=https://yourdomain.funwhale +# Configure email sending using this variale +# By default, funkwhale will output emails sent to stdout +# here are a few examples for this setting +# EMAIL_CONFIG=consolemail:// # output emails to console (the default) +# EMAIL_CONFIG=dummymail:// # disable email sending completely +# On a production instance, you'll usually want to use an external SMTP server: +# EMAIL_CONFIG=smtp://user@:password@youremail.host:25' +# EMAIL_CONFIG=smtp+ssl://user@:password@youremail.host:465' +# EMAIL_CONFIG=smtp+tls://user@:password@youremail.host:587' + +# The email address to use to send systme emails. By default, we will +# DEFAULT_FROM_EMAIL=noreply@yourdomain + # Depending on the reverse proxy used in front of your funkwhale instance, # the API will use different kind of headers to serve audio files # Allowed values: nginx, apache2 @@ -88,9 +102,6 @@ DJANGO_SECRET_KEY= # want to # DJANGO_ADMIN_URL=^api/admin/ -# If True, unauthenticated users won't be able to query the API -API_AUTHENTICATION_REQUIRED=True - # Sentry/Raven error reporting (server side) # Enable Raven if you want to help improve funkwhale by # automatically sending error reports our Sentry instance. @@ -98,15 +109,6 @@ API_AUTHENTICATION_REQUIRED=True RAVEN_ENABLED=false RAVEN_DSN=https://44332e9fdd3d42879c7d35bf8562c6a4:0062dc16a22b41679cd5765e5342f716@sentry.eliotberriot.com/5 -# This settings enable/disable federation on the instance level -FEDERATION_ENABLED=True -# This setting decide wether music library is shared automatically -# to followers or if it requires manual approval before. -# FEDERATION_MUSIC_NEEDS_APPROVAL=False -# means anyone can subscribe to your library and import your file, -# use with caution. -FEDERATION_MUSIC_NEEDS_APPROVAL=True - # In-place import settings # You can safely leave those settings uncommented if you don't plan to use # in place imports. diff --git a/deploy/funkwhale-server.service b/deploy/funkwhale-server.service index 53d3a104b..88d70d338 100644 --- a/deploy/funkwhale-server.service +++ b/deploy/funkwhale-server.service @@ -8,7 +8,7 @@ User=funkwhale # adapt this depending on the path of your funkwhale installation WorkingDirectory=/srv/funkwhale/api EnvironmentFile=/srv/funkwhale/config/.env -ExecStart=/srv/funkwhale/virtualenv/bin/daphne -b ${FUNKWHALE_API_IP} -p ${FUNKWHALE_API_PORT} config.asgi:application +ExecStart=/srv/funkwhale/virtualenv/bin/daphne -b ${FUNKWHALE_API_IP} -p ${FUNKWHALE_API_PORT} config.asgi:application --proxy-headers [Install] WantedBy=multi-user.target diff --git a/dev.yml b/dev.yml index 264fc9534..534d8f5b5 100644 --- a/dev.yml +++ b/dev.yml @@ -123,6 +123,15 @@ services: - '35730:35730' - '8001:8001' + api-docs: + image: swaggerapi/swagger-ui + environment: + - "API_URL=/swagger.yml" + ports: + - '8002:8080' + volumes: + - "./api/docs/swagger.yml:/usr/share/nginx/html/swagger.yml" + networks: internal: federation: diff --git a/docs/api.rst b/docs/api.rst new file mode 100644 index 000000000..650b3885e --- /dev/null +++ b/docs/api.rst @@ -0,0 +1,6 @@ +Funkwhale API +============= + +Funkwhale API is still a work in progress and should not be considered as +stable. We offer an `interactive documentation using swagger `_ +were you can browse available endpoints and try the API. diff --git a/docs/build_docs.sh b/docs/build_docs.sh new file mode 100755 index 000000000..fbf2036af --- /dev/null +++ b/docs/build_docs.sh @@ -0,0 +1,5 @@ +#!/bin/bash -eux +# Building sphinx and swagger docs + +python -m sphinx . $BUILD_PATH +TARGET_PATH="$BUILD_PATH/swagger" ./build_swagger.sh diff --git a/docs/build_swagger.sh b/docs/build_swagger.sh new file mode 100755 index 000000000..13ae21b06 --- /dev/null +++ b/docs/build_swagger.sh @@ -0,0 +1,9 @@ +#!/bin/bash -eux + +SWAGGER_VERSION="3.13.6" +TARGET_PATH=${TARGET_PATH-"swagger"} +rm -rf $TARGET_PATH /tmp/swagger-ui +git clone --branch="v$SWAGGER_VERSION" --depth=1 "https://github.com/swagger-api/swagger-ui.git" /tmp/swagger-ui +mv /tmp/swagger-ui/dist $TARGET_PATH +cp swagger.yml $TARGET_PATH +sed -i "s,http://petstore.swagger.io/v2/swagger.json,swagger.yml,g" $TARGET_PATH/index.html diff --git a/docs/configuration.rst b/docs/configuration.rst index c0de76f56..bbc658e08 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -18,6 +18,8 @@ and technical aspects of your instance, such as database credentials. on environment variables. +.. _instance-settings: + Instance settings ----------------- @@ -37,6 +39,38 @@ settings in this interface. Configuration reference ----------------------- +.. _setting-EMAIL_CONFIG: + +``EMAIL_CONFIG`` +^^^^^^^^^^^^^^^^ + +Determine how emails are sent. + +Default: ``consolemail://`` + +Possible values: + +- ``consolemail://``: Output sent emails to stdout +- ``dummymail://``: Completely discard sent emails +- ``smtp://user:password@youremail.host:25``: Send emails via SMTP via youremail.host on port 25, without encryption, authenticating as user "user" with password "password" +- ``smtp+ssl://user:password@youremail.host:465``: Send emails via SMTP via youremail.host on port 465, using SSL encryption, authenticating as user "user" with password "password" +- ``smtp+tls://user:password@youremail.host:587``: Send emails via SMTP via youremail.host on port 587, using TLS encryption, authenticating as user "user" with password "password" + +.. _setting-DEFAULT_FROM_EMAIL: + +``DEFAULT_FROM_EMAIL`` +^^^^^^^^^^^^^^^^^^^^^^ + +The email address to use to send email. + +Default: ``Funkwhale `` + +.. note:: + + Both the forms ``Funkwhale `` and + ``noreply@yourdomain`` work. + + .. _setting-MUSIC_DIRECTORY_PATH: ``MUSIC_DIRECTORY_PATH`` diff --git a/docs/contributing.rst b/docs/contributing.rst new file mode 100644 index 000000000..0e5a08ecf --- /dev/null +++ b/docs/contributing.rst @@ -0,0 +1 @@ +.. include:: ../CONTRIBUTING diff --git a/docs/federation.rst b/docs/federation.rst index 5b030074c..0f016ada9 100644 --- a/docs/federation.rst +++ b/docs/federation.rst @@ -12,8 +12,7 @@ Managing federation Federation management is only available to instance admins and users who have the proper permissions. You can disable federation completely -at the instance level by setting the FEDERATION_ENABLED environment variable -to False. +at the instance level by editing the ``federation__enabled`` :ref:`setting `. On the front end, assuming you have the proper permission, you will see a "Federation" link in the sidebar. @@ -52,6 +51,6 @@ each other instance asking for access to library. This is by design, to ensure your library is not shared publicly without your consent. However, if you're confident about federating publicly without manual approval, -you can set the FEDERATION_MUSIC_NEEDS_APPROVAL environment variable to false. +you can set the ``federation__music_needs_approval`` :ref:`setting ` to false. Follow requests will be accepted automatically and followers given access to your library without manual intervention. diff --git a/docs/index.rst b/docs/index.rst index 82dcf8c88..481690b70 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,7 +16,10 @@ Funkwhale is a self-hosted, modern free and open-source music server, heavily in configuration importing-music federation + api upgrading + third-party + contributing changelog Indices and tables diff --git a/docs/installation/debian.rst b/docs/installation/debian.rst index c4e54218d..eb0c3f0ea 100644 --- a/docs/installation/debian.rst +++ b/docs/installation/debian.rst @@ -31,7 +31,7 @@ Layout All funkwhale-related files will be located under ``/srv/funkwhale`` apart from database files and a few configuration files. We will also have a -dedicated ``funwhale`` user to launch the processes we need and own those files. +dedicated ``funkwhale`` user to launch the processes we need and own those files. You are free to use different values here, just remember to adapt those in the next steps. diff --git a/docs/installation/external_dependencies.rst b/docs/installation/external_dependencies.rst index 6641bef00..39d32b38f 100644 --- a/docs/installation/external_dependencies.rst +++ b/docs/installation/external_dependencies.rst @@ -18,7 +18,7 @@ On debian-like systems, you would install the database server like this: .. code-block:: shell - sudo apt-get install postgresql + sudo apt-get install postgresql postgresql-contrib The remaining steps are heavily inspired from `this Digital Ocean guide `_. @@ -32,13 +32,22 @@ Create the project database and user: .. code-block:: shell - CREATE DATABASE funkwhale; + CREATE DATABASE "scratch" + WITH ENCODING 'utf8' + LC_COLLATE = 'en_US.utf8' + LC_CTYPE = 'en_US.utf8'; CREATE USER funkwhale; GRANT ALL PRIVILEGES ON DATABASE funkwhale TO funkwhale; Assuming you already have :ref:`created your funkwhale user `, you should now be able to open a postgresql shell: +.. warning:: + + It's importing that you use utf-8 encoding for your database, + otherwise you'll end up with errors and crashes later on when dealing + with music metedata that contains non-ascii chars. + .. code-block:: shell sudo -u funkwhale -H psql @@ -49,7 +58,7 @@ for funkwhale to work properly: .. code-block:: shell - sudo -u postgres psql -c 'CREATE EXTENSION "unaccent";'' + sudo -u postgres psql -c 'CREATE EXTENSION "unaccent";' Cache setup (Redis) diff --git a/docs/installation/index.rst b/docs/installation/index.rst index 776c22424..a3e11529b 100644 --- a/docs/installation/index.rst +++ b/docs/installation/index.rst @@ -28,10 +28,16 @@ On a dockerized instance with 2 CPUs and a few active users, the memory footprin funkwhale_postgres_1 22.73 MiB funkwhale_redis_1 1.496 MiB +Some users have reported running Funkwhale on Raspberry Pis with a memory +consumption of less than 350MiB. + Thus, Funkwhale should run fine on commodity hardware, small hosting boxes and Raspberry Pi. We lack real-world exemples of such deployments, so don't hesitate do give us your feedback (either positive or negative). +Check out :doc:`optimization` for advices on how to tune your instance on small +configurations. + Software requirements --------------------- @@ -86,7 +92,7 @@ Files for the web frontend are purely static and can simply be downloaded, unzip Reverse proxy -------------- -In order to make funkwhale accessible from outside your server and to play nicely with other applications on your machine, you should configure a reverse proxy. At the moment, we only have documentation for nginx, if you know how to implement the same thing for apache, you're welcome. +In order to make funkwhale accessible from outside your server and to play nicely with other applications on your machine, you should configure a reverse proxy. Nginx ^^^^^ @@ -103,9 +109,44 @@ Then, download our sample virtualhost file and proxy conf: .. parsed-literal:: curl -L -o /etc/nginx/funkwhale_proxy.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/funkwhale_proxy.conf" - curl -L -o /etc/nginx/sites-enabled/funkwhale.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/nginx.conf" + curl -L -o /etc/nginx/sites-available/funkwhale.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/nginx.conf" + ln -s /etc/nginx/sites-available/funkwhale.conf /etc/nginx/sites-enabled/ -Ensure static assets and proxy pass match your configuration, and check the configuration is valid with ``nginx -t``. If everything is fine, you can restart your nginx server with ``service nginx restart``. +Ensure static assets and proxy pass match your configuration, and check the configuration is valid with ``nginx -t``. +If everything is fine, you can restart your nginx server with ``service nginx restart``. + +Apache2 +^^^^^^^ + +.. note:: + + Apache2 support is still very recent and the following features + are not working yet: + + - Websocket (used for real-time updates on Instance timeline) + - Transcoding of audio files + + Those features are not necessary to use your Funkwhale instance, and + transcoding in particular is still in alpha-state anyway. + +Ensure you have a recent version of apache2 installed on your server. +You'll also need the following dependencies:: + + apt install libapache2-mod-xsendfile + +Then, download our sample virtualhost file: + +.. parsed-literal:: + + curl -L -o /etc/apache2/sites-available/funkwhale.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/apache.conf" + ln -s /etc/apache2/sites-available/funkwhale.conf /etc/apache2/sites-enabled/ + +You can tweak the configuration file according to your setup, especially the +TLS configuration. Otherwise, defaults, should work if you followed the +installation guide. + +Check the configuration is valid with ``apache2ctl configtest``, and once you're +done, load the new configuration with ``service apache2 restart``. About internal locations ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/installation/optimization.rst b/docs/installation/optimization.rst new file mode 100644 index 000000000..f873795e2 --- /dev/null +++ b/docs/installation/optimization.rst @@ -0,0 +1,37 @@ +Optimizing your Funkwhale instance +================================== + +Depending on your requirements, you may want to reduce as much as possible +Funkwhale's footprint. + +Reduce workers concurrency +-------------------------- + +Asynchronous tasks are handled by a celery worker, which will by default +spawn a worker process per CPU available. This can lead to a higher +memory usage. + +You can control this behaviour using the ``--concurrency`` flag. +For instance, setting ``--concurrency=1`` will spawn only one worker. + +This flag should be appended after the ``celery -A funkwhale_api.taskapp worker`` +command in your :file:`docker-compose.yml` file if your using Docker, or in your +:file:`/etc/systemd/system/funkwhale-worker.service` otherwise. + +.. note:: + + Reducing concurrency comes at a cost: asynchronous tasks will be processed + more slowly. However, on small instances, this should not be an issue. + + +Switch from prefork to solo pool +-------------------------------- + +Using a different pool implementation for Celery tasks may also help. + +Using the ``solo`` pool type should reduce your memory consumption. +You can control this behaviour using the ``--pool=solo`` flag. + +This flag should be appended after the ``celery -A funkwhale_api.taskapp worker`` +command in your :file:`docker-compose.yml` file if your using Docker, or in your +:file:`/etc/systemd/system/funkwhale-worker.service` otherwise. diff --git a/docs/swagger.yml b/docs/swagger.yml new file mode 100644 index 000000000..7735a8f20 --- /dev/null +++ b/docs/swagger.yml @@ -0,0 +1,186 @@ +openapi: "3.0" +info: + description: "Documentation for [Funkwhale](https://funkwhale.audio) API. The API is **not** stable yet." + version: "1.0.0" + title: "Funkwhale API" + +servers: + - url: https://demo.funkwhale.audio/api/v1 + description: Demo server + - url: https://node1.funkwhale.test/api/v1 + description: Node 1 (local) + +components: + securitySchemes: + jwt: + type: http + scheme: bearer + bearerFormat: JWT + description: "You can get a token by using the /token endpoint" + +security: + - jwt: [] + +paths: + /token/: + post: + tags: + - "auth" + description: + Obtain a JWT token you can use for authenticating your next requests. + security: [] + responses: + '200': + description: Successfull auth + '400': + description: Invalid credentials + requestBody: + required: true + content: + application/json: + schema: + type: "object" + properties: + username: + type: "string" + example: "demo" + password: + type: "string" + example: "demo" + + /artists/: + get: + tags: + - "artists" + parameters: + - name: "q" + in: "query" + description: "Search query used to filter artists" + schema: + required: false + type: "string" + example: "carpenter" + - name: "listenable" + in: "query" + description: "Filter/exclude artists with listenable tracks" + schema: + required: false + type: "boolean" + responses: + 200: + content: + application/json: + schema: + type: "object" + properties: + count: + $ref: "#/properties/resultsCount" + results: + type: "array" + items: + $ref: "#/definitions/ArtistNested" + +properties: + resultsCount: + type: "integer" + format: "int64" + description: "The total number of resources matching the request" + mbid: + type: "string" + formats: "uuid" + description: "A musicbrainz ID" +definitions: + Artist: + type: "object" + properties: + mbid: + required: false + $ref: "#/properties/mbid" + id: + type: "integer" + format: "int64" + example: 42 + name: + type: "string" + example: "System of a Down" + creation_date: + type: "string" + format: "date-time" + ArtistNested: + type: "object" + allOf: + - $ref: "#/definitions/Artist" + - type: "object" + properties: + albums: + type: "array" + items: + $ref: "#/definitions/AlbumNested" + + Album: + type: "object" + properties: + mbid: + required: false + $ref: "#/properties/mbid" + id: + type: "integer" + format: "int64" + example: 16 + artist: + type: "integer" + format: "int64" + example: 42 + title: + type: "string" + example: "Toxicity" + creation_date: + type: "string" + format: "date-time" + release_date: + type: "string" + required: false + format: "date" + example: "2001-01-01" + + AlbumNested: + type: "object" + allOf: + - $ref: "#/definitions/Album" + - type: "object" + properties: + tracks: + type: "array" + items: + $ref: "#/definitions/Track" + + Track: + type: "object" + properties: + mbid: + required: false + $ref: "#/properties/mbid" + id: + type: "integer" + format: "int64" + example: 66 + artist: + type: "integer" + format: "int64" + example: 42 + album: + type: "integer" + format: "int64" + example: 16 + title: + type: "string" + example: "Chop Suey!" + position: + required: false + description: "Position of the track in the album" + type: "number" + minimum: 1 + example: 1 + creation_date: + type: "string" + format: "date-time" diff --git a/docs/third-party.rst b/docs/third-party.rst new file mode 100644 index 000000000..0335f8c71 --- /dev/null +++ b/docs/third-party.rst @@ -0,0 +1,17 @@ +Third party projects +==================== + +This page lists all known projects that are maintained by third-parties +and integrate or relates to Funkwhale. + +.. note:: + + If you want your project to be added or removed from this page, + please open an issue on our issue tracker. + + +API Clients +----------- + +- `libfunkwhale `_: a Funkwhale API written in Vala +- `Funkwhale-javalib `_: a Funkwhale API client written in Java diff --git a/front/src/App.vue b/front/src/App.vue index e8cac7476..a21337428 100644 --- a/front/src/App.vue +++ b/front/src/App.vue @@ -97,6 +97,12 @@ html, body { } } +.ellipsis { + text-overflow: ellipsis; + white-space: nowrap; + overflow: hidden; +} + .ui.small.text.container { max-width: 500px !important; } diff --git a/front/src/assets/logo/logo-full-500.png b/front/src/assets/logo/logo-full-500.png new file mode 100644 index 000000000..952f38033 Binary files /dev/null and b/front/src/assets/logo/logo-full-500.png differ diff --git a/front/src/components/Sidebar.vue b/front/src/components/Sidebar.vue index fb4074d80..97c743bbe 100644 --- a/front/src/components/Sidebar.vue +++ b/front/src/components/Sidebar.vue @@ -30,24 +30,56 @@
-
- {{ $t('Logged in as {%name%}', { name: $store.state.auth.username }) }} - {{ $t('Logout') }} - {{ $t('Login') }} - {{ $t('Browse library') }} - {{ $t('Favorites') }} - - {{ $t('Playlists') }} - - {{ $t('Activity') }} - {{ $t('Federation') }} +
+
+
{{ $t('My account') }}
+
+ {{ $t('Logged in as {%name%}', { name: $store.state.auth.username }) }} + {{ $t('Logout') }} + {{ $t('Login') }} +
+
+
+
{{ $t('Music') }}
+
+ {{ $t('Browse library') }} + {{ $t('Favorites') }} + + {{ $t('Playlists') }} + + {{ $t('Activity') }} +
+
+
+
{{ $t('Administration') }}
+
+ + {{ $t('Import requests') }} +
+ {{ notifications.importRequests }}
+ + + {{ $t('Federation') }} +
+ {{ notifications.federation }}
+ +
+
@@ -104,6 +136,7 @@ diff --git a/front/src/components/federation/LibraryCard.vue b/front/src/components/federation/LibraryCard.vue index 757561fb3..e7ef7a516 100644 --- a/front/src/components/federation/LibraryCard.vue +++ b/front/src/components/federation/LibraryCard.vue @@ -1,8 +1,14 @@ diff --git a/front/src/components/library/import/BatchDetail.vue b/front/src/components/library/import/BatchDetail.vue index b73c8cf82..f0e6502f0 100644 --- a/front/src/components/library/import/BatchDetail.vue +++ b/front/src/components/library/import/BatchDetail.vue @@ -40,7 +40,16 @@ {{ $t('Errored') }} - {{ stats.errored }} + + {{ stats.errored }} + + {{ $t('Finished') }} @@ -83,11 +92,21 @@ {{ job.mbid }} - {{ job.source }} + + {{ job.source|truncate(50) }} + {{ job.status }} + :class="['ui', {'yellow': job.status === 'pending'}, {'red': job.status === 'errored'}, {'green': job.status === 'finished'}, 'label']"> + {{ job.status }} + {{ job.track_file.track }} @@ -167,12 +186,6 @@ export default { return axios.get(url).then((response) => { self.batch = response.data self.isLoading = false - if (self.batch.status === 'pending') { - self.timeout = setTimeout( - self.fetchData, - 5000 - ) - } }) }, fetchStats () { @@ -186,7 +199,7 @@ export default { self.fetchJobs() self.fetchData() } - if (self.batch.status === 'pending') { + if (self.stats.pending > 0) { self.timeout = setTimeout( self.fetchStats, 5000 @@ -194,6 +207,15 @@ export default { } }) }, + rerun ({jobs, batches}) { + let payload = { + jobs, batches + } + let self = this + axios.post('import-jobs/run/', payload).then((response) => { + self.fetchStats() + }) + }, fetchJobs () { let params = { batch: this.id, diff --git a/front/src/components/requests/RequestsList.vue b/front/src/components/requests/RequestsList.vue index 4464031c5..130214c3a 100644 --- a/front/src/components/requests/RequestsList.vue +++ b/front/src/components/requests/RequestsList.vue @@ -8,6 +8,16 @@
+
+ + +
+
@@ -115,6 +115,7 @@ export default { }, mounted () { $('.ui.dropdown').dropdown() + $(this.$el).find('.field .search').focus() }, methods: { updateQueryString: _.debounce(function () {