The `WEB_CONCURRENCY` environment variable is a more common way to
configure the number of workers for Gunicorn [1] or other web servers.
This change maintains `GUNICORN_WORKERS` compatibility, while notifying
users that it is deprecated and should be replaced with
`WEB_CONCURRENCY`.
It would also allow us to replace Gunicorn with another web server in
the future without changing the variable name.
[1] https://docs.gunicorn.org/en/stable/settings.html#workers
Using the `envsubst` command, we can replace environment variables in
the nginx template files. This allows for more flexibility when
configuring the nginx server.
The Docker image we use as base for Nginx does provide the
`20-envsubst-on-templates.sh` script that will replace environment
variables in the template files.
This change does not include any behavior change, but unblocks future
changes that require environment variables in the nginx configuration.
This change removes the installation of `dev` and `test` Poetry
dependency groups from the published Docker images.
Developers are still able to use images with development dependencies
installed, by either using the `dev-slim` or `dev-full` targets.
Image size comparison:
* `slim-image`: Down from 455 MiB to 355 MiB.
* `full-image`: Down from 760 MiB to 660 MiB.
This change only verifies that Alembic can upgrade and downgrade through
all the current migrations. It does not verify that the application
works correctly with PostgreSQL.
This commit improves Docker layer caching by installing the required
packages in the `production-stage` stage, before copying files from
other stages.
This way, dependency installation will remain cached unless there's a
change in the required `apk` packages (or the stage's base image).
`tini` does not wait for child processes to close, so all processes will be killed immediately. This is why the container stops so fast.
This fix makes the `init` script listen and handle terminate signals. It also ensures that child processes are shut down in reverse order with proper waiting for completion.
This change allows setting environment variables with a `_FILE` suffix,
which will be used to load the contents of the file specified in the
variable into the variable without the suffix.
For example, setting `ROMM_AUTH_SECRET_KEY_FILE=/run/secrets/romm_auth_secret_key`
and creating a file with the secret key at the specified path will set
`ROMM_AUTH_SECRET_KEY` to the contents of the file.
A common use case for this is to use secrets in Docker Compose [1], to
avoid exposing secrets in the `docker-compose.yml` or `env` files.
[1] https://docs.docker.com/compose/how-tos/use-secrets/
At the moment, 7zip files are generating memory issues and even OOM
errors on user installations. This is because the current stable release
of `py7zr` does not support decompression streaming, and RomM needs to
decompress the each 7zip file in the library into memory to be able to
calculate hashes.
This change introduces a `py7zr` fork I created to have a stable commit
SHA to refer to in case upstream gets any forced pushes. It includes the
contents of the pull request the `py7zr` creator is working on to
support decompression streaming [1].
The way decompression streaming is implemented in `py7zr` is different
than the other compression utilities. Instead of being able to provide a
`bytes` iterator, we need to provide a `Py7zIO` implementation that
will call a callback on each read and write operation.
[1] https://github.com/miurahr/py7zr/pull/620
Use tini [1] as the container entrypoint to ensure that the container
process is properly reaped when the container is stopped.
This is needed as the main container command is a shell script.
A simple way to test this change is to:
1. Build the container image and run it using `docker compose up`.
2. Within the same terminal, stop the container using `Ctrl+C`.
3. Verify that the container is properly stopped right away.
Without `tini`, the container takes a few seconds to stop, and it makes
one CPU core to be used at 100% during that time.
[1] https://github.com/krallin/tini
Build and include the `RAHasher` binary in the Docker image, to
calculate hashes for RetroAchievements. Also, add a service to
run `RAHasher` from Python.
Example usage:
```python
from adapters.services.rahasher import RAHasherError, RAHasherService
rahasher = RAHasherService()
try:
hash = await rahasher.calculate_hash("nes", Path("path/to/rom.nes"))
except RAHasherError:
# Handle error
hash = None
```
This change replaces the bundled Redis server with Valkey. No breaking
changes are introduced, as considered environment variables still
maintain the `REDIS_` prefix.
Fixes#925.
This change installs and configures the `mod_zip` nginx module [1],
which allows nginx to stream ZIP files directly.
It includes a workaround needed to correctly calculate CRC-32 values for
included files, by including a new `server` section listening at port
8081, only used for the file requests to be upstream subrequests that
correctly trigger the CRC-32 calculation logic.
Also, to be able to provide a `m3u` file generated on the fly, we add a
`/decode` endpoint fully implemented in nginx using NJS, which receives
a `value` URL param, and decodes it using base64. The decoded value is
returned as the response.
That way, the contents of the `m3u` file is base64-encoded, and set as
part of the response, for `mod_zip` to include it in the ZIP file.
[1] https://github.com/evanmiller/mod_zip
Instead of making FastAPI handle file download, which has serious
performance issues on big files [1], this change uses nginx's `X-Accel`
feature to delegate single-file downloads to nginx.
Partial fix for #1079, as it solves the CPU usage issue for single-file
downloads.
[1] https://github.com/fastapi/fastapi/discussions/6050
Currently, the `request.url_for` and `URLPath.make_absolute_url` methods
always build URLs with "http" scheme, even when the original requested
URL is using "https".
The reason for this is that Gunicorn does not allow IPs other than
127.0.0.1 to set secure headers by default. As regular RomM
installations don't know which frontend IPs will try to set security
headers in advance, we can disable this validation, and fix URL
building.
A simple way to test this change is to access any of the `feed` endpoints,
which generate URLs using the mentioned methods. Accessing the endpoint
using "https" scheme must generate "https" URLs.
Reference:
* https://github.com/encode/starlette/issues/538#issuecomment-2054013679
* https://docs.gunicorn.org/en/stable/settings.html#forwarded-allow-ips
Fix FastAPI and nginx configuration, to make the application correctly
redirect URLs. This is specially useful when URLs ended with forward
slash are redirected to their stripped version.
Included changes:
* Stop removing the `/api` prefix in nginx rewrite rules, so FastAPI
knows what's the original URL path being requested.
* Use `$http_host` in nginx, so FastAPI receives both the original host
and port, to build the redirect URL (as `$host` does not include the
port, if present).
* Make all FastAPI included routers know their prefix, to correctly
route incoming requests.
This fix was found based on a report that redirects from URLs ended with
forward slash were not working [1].
[1] https://github.com/rommapp/romm/issues/1051#issuecomment-2269049762
