This improvement avoids extra IGDB API requests when a received Rom is
an exact match. It avoids up to 2 requests per Rom, when an exact match
is found.
Convert `IGDBBaseHandler` methods to be asynchronous, and use an `httpx`
async client, instead of `requests` sync client.
This change also removes the direct dependency with `requests`, as the
project no longer uses it, preferring `httpx` instead.
This change mainly refactors the `scan_platforms` function, moving part
of its logic to `_identify_platform`, `_identify_firmware`, and
`_identify_rom`.
The logic is simpler this way, each smaller function returns `ScanStats`
that can be merged by the caller, and it simplifies future performance
improvements.
For filesystem resource handler, `requests` calls have been replaced
with `httpx`, and file I/O has been replaced with `anyio` utils.
The existing approach to save covers and screenshots, by calling
`shutil.copyfileobj` with the raw response is no longer needed. `httpx`
does not provide a file-like object when streaming [1], so there's no
easy drop-in replacement.
However, the applied solution correctly builds the file iteratively, by
consuming the response in chunks.
[1] https://github.com/encode/httpx/discussions/2296
Introduce an asynchronous Redis instance to be used in async functions.
Also, this change migrates most of the sync cache usage to the new async
cache.
`os.walk` is a generator that can iteratively navigate from the
specified path, top-bottom. However, most of the calls to `os.walk` in
the project cast the call to `list()`, which makes it traverse the path
and recursively find all nested directories.
This is commonly not needed, as we end up just using a `[0]` index to
only access the root path.
This change adds a few utils that simplifies listing files/directories,
and by default does it non-recursively. Performance gains shouldn't be
noticeable in systems with high-speed storage, but we can avoid the edge
cases of users having too many nested directories, by avoiding unneeded
I/O.
This change avoids blocking requests when retrieving covers from
SteamGridDB, which is the main bottleneck as the current implementation
iterates over paginated results for multiple games.
Using an asynchronous client like `httpx` provides a good performance
improvement, and reduces the latency when calling this endpoint.
Also, the inclusion of FastAPI `lifespan` allows instantiating a single
client on startup.
When testing with "Final Fantasy V Advance", the endpoint goes from ~9s
to ~1.5s to retrieve all covers.
The SteamGridDB API has a bug, where the `total` field for the paginated
endpoint `/v2/grids/game/:gameId` sometimes double/triple counts results
(e.g. returning `total=172` when there are actually 86 results).
To avoid this issue, this change relies on each page's number of
results. If the received page has less than the requested amount of
grids, we know that is the last page.
The current implementation for some of the database handlers, where the
same method is used to retrieve either a single entity (when an `id` is
passed), a list of entities, or `None`, makes the typing and overall
design more complex.
This change simplifies database handlers, by having two separate methods
where appropiate:
* A method that receives an `id`, and returns either an entity, or `None`.
* A method that optionally receives filters, and returns (depending on
the current handler implementation) a list of entities, or a `Select`
object that allows chaining more SQLAlchemy operations.
