I loaded data from Apple Music as a one-off into the duckdb instance -- but it has been months since I refreshed it. It's time to make this a bona-fide service that we can manage and monitor etc.
In general, the iTunes API has pretty low quota limits so hitting it for a backfill is not usually going to work. And my local laptop is not the right place to have a persistent server, that holds the entirety of the apple database, and the duckdb for serving queries.
Success is:
- a new service on Render, sized sufficiently to hold the Apple Music database (including all of the necessary supporting tables, images, artist names, etc.)
- a job that can be invoked manually or on a schedule to refresh the Apple Music database with one-click invocation, including management, logging, and reporting of problems
- an update to the apple match logic to query the new 'production' Apple Music database,
- only as a last resort, relying on the iTunes API since the query limits are so low.
When we tackled this originally, we decided on duckDB -- but I am open to revisiting that idea; I'm looking for maximum ease of management; I'm willing to pay a little bit for disk or server to make this easier and more efficient,.
I loaded data from Apple Music as a one-off into the duckdb instance -- but it has been months since I refreshed it. It's time to make this a bona-fide service that we can manage and monitor etc.
In general, the iTunes API has pretty low quota limits so hitting it for a backfill is not usually going to work. And my local laptop is not the right place to have a persistent server, that holds the entirety of the apple database, and the duckdb for serving queries.
Success is:
When we tackled this originally, we decided on duckDB -- but I am open to revisiting that idea; I'm looking for maximum ease of management; I'm willing to pay a little bit for disk or server to make this easier and more efficient,.