At the moment, 7zip files are generating memory issues and even OOM
errors on user installations. This is because the current stable release
of `py7zr` does not support decompression streaming, and RomM needs to
decompress the each 7zip file in the library into memory to be able to
calculate hashes.
This change introduces a `py7zr` fork I created to have a stable commit
SHA to refer to in case upstream gets any forced pushes. It includes the
contents of the pull request the `py7zr` creator is working on to
support decompression streaming [1].
The way decompression streaming is implemented in `py7zr` is different
than the other compression utilities. Instead of being able to provide a
`bytes` iterator, we need to provide a `Py7zIO` implementation that
will call a callback on each read and write operation.
[1] https://github.com/miurahr/py7zr/pull/620
The short-term goal is to completely typehint the IGDB API responses. This
first change adds the base structures and enums RomM currently uses.
The `ExpandableField` type will allow us to model the expansion
mechanism the IGDB API provides, where a field can include either an ID,
or the full nested structure.
This change replaces the creation of Redis URL, from a simple string
interpolation, to using `yarl.URL`. The main benefit, besides not
forgetting to set all five different variables on every Redis client
initialization, is that user credentials are correctly URL-encoded, if
present.
Up until now, if a password had special characters, it could break the
generated URL.
This change also introduces support for a `REDIS_SSL` setting, which
allows the user to specify if the Redis connection should use SSL or not.
This change improves memory usage, by only keeping a single archive's
member file in memory at a time during 7zip decompression.
The `py7zr` library does not support streaming decompression yet, so
this change is the best we can do for now.
Potential fix for #1211, but it won't improve memory usage for
single-file 7zip archives.
This change installs and configures the `mod_zip` nginx module [1],
which allows nginx to stream ZIP files directly.
It includes a workaround needed to correctly calculate CRC-32 values for
included files, by including a new `server` section listening at port
8081, only used for the file requests to be upstream subrequests that
correctly trigger the CRC-32 calculation logic.
Also, to be able to provide a `m3u` file generated on the fly, we add a
`/decode` endpoint fully implemented in nginx using NJS, which receives
a `value` URL param, and decodes it using base64. The decoded value is
returned as the response.
That way, the contents of the `m3u` file is base64-encoded, and set as
part of the response, for `mod_zip` to include it in the ZIP file.
[1] https://github.com/evanmiller/mod_zip
Pytest v8.2 introduced the `PYTEST_VERSION` environment variable [1],
that can be used to check if code is running from within a pytest run.
This way, we can avoid checking the loaded `sys` modules.
[1] https://docs.pytest.org/en/stable/changelog.html#id57
Convert `IGDBBaseHandler` methods to be asynchronous, and use an `httpx`
async client, instead of `requests` sync client.
This change also removes the direct dependency with `requests`, as the
project no longer uses it, preferring `httpx` instead.
For filesystem resource handler, `requests` calls have been replaced
with `httpx`, and file I/O has been replaced with `anyio` utils.
The existing approach to save covers and screenshots, by calling
`shutil.copyfileobj` with the raw response is no longer needed. `httpx`
does not provide a file-like object when streaming [1], so there's no
easy drop-in replacement.
However, the applied solution correctly builds the file iteratively, by
consuming the response in chunks.
[1] https://github.com/encode/httpx/discussions/2296
This change applies the guided migration process recommended by
SQLAlchemy [1], up to step 4, to have declarative ORM models that better
support Python typing.
The change was tested by running `alembic check`, which does not find
any schema changes.
Errors reported by `mypy` go down to 170, from the original 223 in the
current `master` commit.
[1] https://docs.sqlalchemy.org/en/20/changelog/whatsnew_20.html#migrating-an-existing-mapping
`aioredis` is abandoned, and has been replaced by `redis.asyncio`.
Currently, the project doesn't use `aioredis` directly, but depends on
it for `socketio.AsyncRedisManager`. This change is safe, as `socketio`
has been prioritizing `redis.asyncio` over `aioredis` since `v5.6.0` [1].
[1] 0e7691b776
