At the moment, 7zip files are generating memory issues and even OOM
errors on user installations. This is because the current stable release
of `py7zr` does not support decompression streaming, and RomM needs to
decompress the each 7zip file in the library into memory to be able to
calculate hashes.
This change introduces a `py7zr` fork I created to have a stable commit
SHA to refer to in case upstream gets any forced pushes. It includes the
contents of the pull request the `py7zr` creator is working on to
support decompression streaming [1].
The way decompression streaming is implemented in `py7zr` is different
than the other compression utilities. Instead of being able to provide a
`bytes` iterator, we need to provide a `Py7zIO` implementation that
will call a callback on each read and write operation.
[1] https://github.com/miurahr/py7zr/pull/620
When serving files using the `X-Accel-Redirect` header in nginx, the
header values must be URL-encoded. Otherwise, nginx will not be able
to serve the files if they contain special characters.
This commit adds a new `FileRedirectResponse` class to the `utils.nginx`
module, to simplify the creation of responses that serve files using the
`X-Accel-Redirect` header.
Fixes#1212, #1223.
This change installs and configures the `mod_zip` nginx module [1],
which allows nginx to stream ZIP files directly.
It includes a workaround needed to correctly calculate CRC-32 values for
included files, by including a new `server` section listening at port
8081, only used for the file requests to be upstream subrequests that
correctly trigger the CRC-32 calculation logic.
Also, to be able to provide a `m3u` file generated on the fly, we add a
`/decode` endpoint fully implemented in nginx using NJS, which receives
a `value` URL param, and decodes it using base64. The decoded value is
returned as the response.
That way, the contents of the `m3u` file is base64-encoded, and set as
part of the response, for `mod_zip` to include it in the ZIP file.
[1] https://github.com/evanmiller/mod_zip
* Add typing for nested objects in Tinfoil and Webrcade feed schemas.
* Do not send Rom background and thumbnail, if not available.
* Correctly build URLs using `starlette` utils.
* Deprecate the `ROMM_HOST` setting, no longer needed.
* Fix FastAPI custom router to prefer routes without trailing slash.
* Fix Webrcade background URL pointing to gallery screenshot.
According to multiple FastAPI discussions [1], FastAPI only includes a
built-in mechanism to redirect requests including a trailing slash, to
its variation without slash, using a `307` status code.
This can be an issue when certain clients do not send the same headers
on the redirected request.
This change adds a custom FastAPI `APIRouter`, that registers both route
path variations (with and without trailing slash), while only marking
the path without slash for being included in the OpenAPI schema.
[1] https://github.com/fastapi/fastapi/discussions/7298
`os.walk` is a generator that can iteratively navigate from the
specified path, top-bottom. However, most of the calls to `os.walk` in
the project cast the call to `list()`, which makes it traverse the path
and recursively find all nested directories.
This is commonly not needed, as we end up just using a `[0]` index to
only access the root path.
This change adds a few utils that simplifies listing files/directories,
and by default does it non-recursively. Performance gains shouldn't be
noticeable in systems with high-speed storage, but we can avoid the edge
cases of users having too many nested directories, by avoiding unneeded
I/O.
