diff --git a/FAQ.md b/FAQ.md index 72af5192..ace2e0d2 100644 --- a/FAQ.md +++ b/FAQ.md @@ -867,3 +867,20 @@ $ ./bin/console help ``` For more information, please refer to the [Dockerfile](/Dockerfile). On how we do things to get the tool running. + +--- + +### How does the file integrity feature works? + +The feature first scan your entire history for reported media file paths. then we will do stat check on each point of the path starting from lowest to highest. + +For example lets say your media file is `/media/series/season 1/episode 1.mkv` + +We do the following. `/media` exists or not? if it does we move to the next path `/media/series` exists or not? if it does we move to the next path `/media/series/season 1` exists or not? if it does we move to the next path `/media/series/season 1/episode 1.mkv` exists or not? if it does we move to the next path. + +Using this approach allow us to cache calls and reduce unnecessary calls to the filesystem. If you have for example `/media/seriesX/` with thousands of files, +and the root `/media/seriesX` doesn't exists we dont have to call stat for every file, instead using the cache we determine that the file doesn't exist. + +Of course, every stat call is cached, so if 1 or more backends are reporting the same file path, we only do the stat check once. This is to reduce the load on the filesystem. + +--- diff --git a/NEWS.md b/NEWS.md index fe4083be..a5231118 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,43 @@ # Old Updates +### 2024-07-06 + +Recently we have introduced a new feature that allows you to use Jellyfin and Emby OAuth access tokens for syncing +your play state. This is especially handy if you're not the server owner and can't create API keys. Please note, this +feature is in its experimental phase, so you might encounter some issues as we yet to explorer the full depth of the +implementation. We're actively working on making it better, If you have any feedback or suggestions, please let us know. + +Getting your OAuth token is easy. When prompted, simply enter your `username:password` in place of the API key through +the `WebUI` or the `config:add/manage` command. `WatchState` will automatically contact the backend and generate the +token for you, as this step is required to get more information like your `User ID` which is sadly inaccessible without +us generating the token. Both Emby & Jellyfin doesn't provide an API endpoint to inquiry about the current user. + +We have also added new `config:test` command to run functional tests on your backends, this will not alter your state, +And it's quite useful to know if the tool is able to communicate with your backends. without problems, It will report +the following, `OK` which mean the indicated test has passed, `FA` which mean the indicated test has failed. And `SK` +which mean the indicated test has been skipped or not yet implemented. + +### 2024-06-23 + +WE are happy to announce that the `WebUI` is ready for wider usage and we are planning to release it in the next few +months. +We are actively working on it to improve it. If you have any feedback or suggestions, please let us know. We feel it's +almost future complete +for the things that we want. + +On another related news, we have added new environment variable `WS_API_AUTO` "disabled by default" which can be used +to automatically expose your **API KEY/TOKEN**. This is useful for users who are using the `WebUI` from many different +browsers +and want to automate the configuration process. + +While the `WebUI` is included in the main project, it's a standalone feature and requires the API settings to be +configured before it +can be used. This environment variable can be enabled by setting `WS_API_AUTO=true` in `${WS_DATA_PATH}/config/.env`. + +> [!IMPORTANT] +> This environment variable is **GREAT SECURITY RISK**, and we strongly recommend not to use it if `WatchState` is +> exposed to the internet. + ### 2024-05-14 We are happy to announce the beta testing of the `WebUI`. To get started on using it you just need to visit the url `http://localhost:8080` We are supposed to diff --git a/README.md b/README.md index 3580a107..096c5da8 100644 --- a/README.md +++ b/README.md @@ -9,43 +9,20 @@ out of the box, this tool support `Jellyfin`, `Plex` and `Emby` media servers. ## Updates -### 2024-07-06 +### 2024-07-22 -Recently we have introduced a new feature that allows you to use Jellyfin and Emby OAuth access tokens for syncing -your play state. This is especially handy if you're not the server owner and can't create API keys. Please note, this -feature is in its experimental phase, so you might encounter some issues as we yet to explorer the full depth of the -implementation. We're actively working on making it better, If you have any feedback or suggestions, please let us know. +We have recently added a new WebUI feature, `File integrity`, this feature will help you to check if your media backends +are reporting files that are not available on the disk. This feature is still in alpha, and we are working on improving +it. -Getting your OAuth token is easy. When prompted, simply enter your `username:password` in place of the API key through -the `WebUI` or the `config:add/manage` command. `WatchState` will automatically contact the backend and generate the -token for you, as this step is required to get more information like your `User ID` which is sadly inaccessible without -us generating the token. Both Emby & Jellyfin doesn't provide an API endpoint to inquiry about the current user. +This feature `REQUIRES` that you mount your media directories to the `WatchState` container preferably as readonly. There is plans to add +a path replacement feature to allow you change the pathing, but it's not implemented yet. -We have also added new `config:test` command to run functional tests on your backends, this will not alter your state, -And it's quite useful to know if the tool is able to communicate with your backends. without problems, It will report -the following, `OK` which mean the indicated test has passed, `FA` which mean the indicated test has failed. And `SK` -which mean the indicated test has been skipped or not yet implemented. +This feature will work on both local and remote cloud storages provided they are mounted into the container. We also may recommend not to +use this feature depending on how your cloud storage provider treats file stat calls. As it might lead to unnecessary money spending. and of course +it will be slower. -### 2024-06-23 - -WE are happy to announce that the `WebUI` is ready for wider usage and we are planning to release it in the next few -months. -We are actively working on it to improve it. If you have any feedback or suggestions, please let us know. We feel it's -almost future complete -for the things that we want. - -On another related news, we have added new environment variable `WS_API_AUTO` "disabled by default" which can be used -to automatically expose your **API KEY/TOKEN**. This is useful for users who are using the `WebUI` from many different -browsers -and want to automate the configuration process. - -While the `WebUI` is included in the main project, it's a standalone feature and requires the API settings to be -configured before it -can be used. This environment variable can be enabled by setting `WS_API_AUTO=true` in `${WS_DATA_PATH}/config/.env`. - -> [!IMPORTANT] -> This environment variable is **GREAT SECURITY RISK**, and we strongly recommend not to use it if `WatchState` is -> exposed to the internet. +For more information about how we cache the stat calls, please refer to the [FAQ](FAQ.md#How-does-the-file-integrity-feature-works). Refer to [NEWS](NEWS.md) for old updates. diff --git a/frontend/layouts/default.vue b/frontend/layouts/default.vue index 3b69cd7d..dd0891a6 100644 --- a/frontend/layouts/default.vue +++ b/frontend/layouts/default.vue @@ -68,6 +68,11 @@ Data Parity + + + + Files Integrity +
diff --git a/frontend/pages/integrity.vue b/frontend/pages/integrity.vue new file mode 100644 index 00000000..f8be44dd --- /dev/null +++ b/frontend/pages/integrity.vue @@ -0,0 +1,386 @@ + + + diff --git a/frontend/pages/parity.vue b/frontend/pages/parity.vue index 0f172119..50f900a0 100644 --- a/frontend/pages/parity.vue +++ b/frontend/pages/parity.vue @@ -174,29 +174,14 @@
-   - Reported By +   + Has metadata from
- -
-
-
- -
-
-
-   - Not Reported By -
-
- + +
diff --git a/src/API/System/Integrity.php b/src/API/System/Integrity.php new file mode 100644 index 00000000..1daba0a8 --- /dev/null +++ b/src/API/System/Integrity.php @@ -0,0 +1,193 @@ +pdo = $this->db->getPDO(); + } + + /** + * @throws InvalidArgumentException + */ + #[Get(self::URL . '[/]', middleware: [ExceptionHandlerMiddleware::class], name: 'system.integrity')] + public function __invoke(iRequest $request): iResponse + { + if ($this->cache->has('system.integrity')) { + $data = $this->cache->get('system.integrity', []); + $this->dirExists = ag($data, 'dir_exists', []); + $this->checkedFile = ag($data, 'checked_file', []); + $this->fromCache = true; + } + + $response = [ + 'items' => [], + 'fromCache' => $this->fromCache, + ]; + + $sql = "SELECT * FROM state"; + $stmt = $this->db->getPDO()->prepare($sql); + $stmt->execute(); + + $base = Container::get(iState::class); + + foreach ($stmt as $row) { + $entity = $base::fromArray($row); + if (false === $this->checkIntegrity($entity)) { + $response['items'][] = $this->formatEntity($entity, true); + } + } + + $this->cache->set('system.integrity', [ + 'dir_exists' => $this->dirExists, + 'checked_file' => $this->checkedFile, + ], new DateInterval('PT1H')); + + return api_response(HTTP_STATUS::HTTP_OK, $response); + } + + private function checkIntegrity(iState $entity): bool + { + $metadata = $entity->getMetadata(); + + if (empty($metadata)) { + return true; + } + + $checks = []; + + foreach ($metadata as $backend => $data) { + if (!isset($data['path'])) { + continue; + } + + $checks[] = [ + 'backend' => $backend, + 'path' => $data['path'], + 'status' => true, + 'message' => '', + ]; + } + + if (empty($checks)) { + return true; + } + + foreach ($checks as &$check) { + $path = $check['path']; + $dirName = dirname($path); + + if (false === $this->checkPath($dirName)) { + $check['status'] = false; + $check['message'] = "File parent directory does not exist."; + continue; + } else { + $check['status'] = true; + $check['message'] = "File parent directory exists."; + } + + if (false === $this->checkFile($path)) { + $check['status'] = false; + $check['message'] = "File does not exist."; + } else { + $check['status'] = true; + $check['message'] = "File exists."; + } + } + + unset($check); + + foreach ($checks as $check) { + if (false === $check['status']) { + $entity->setContext('integrity', $checks); + return false; + } + } + + return true; + } + + /** + * @throws InvalidArgumentException + */ + #[Delete(self::URL . '[/]', name: 'system.integrity.reset')] + public function resetCache(iRequest $request): iResponse + { + if ($this->cache->has('system.integrity')) { + $this->cache->delete('system.integrity'); + } + + return api_response(HTTP_STATUS::HTTP_OK); + } + + private function checkPath(string $dir): bool + { + $dirs = explode(DIRECTORY_SEPARATOR, $dir); + foreach ($dirs as $i => $dir) { + $path = implode(DIRECTORY_SEPARATOR, array_slice($dirs, 0, $i + 1)); + if (empty($path)) { + continue; + } + if (false === $this->dirExists($path)) { + return false; + } + } + + return true; + } + + private function dirExists(string $dir): bool + { + if (array_key_exists($dir, $this->dirExists)) { + return $this->dirExists[$dir]; + } + + $this->dirExists[$dir] = is_dir($dir); + + return $this->dirExists[$dir]; + } + + private function checkFile(string $file): bool + { + if (array_key_exists($file, $this->checkedFile)) { + return $this->checkedFile[$file]; + } + + $this->checkedFile[$file] = file_exists($file); + + return $this->checkedFile[$file]; + } +} diff --git a/src/Libs/Middlewares/ExceptionHandlerMiddleware.php b/src/Libs/Middlewares/ExceptionHandlerMiddleware.php new file mode 100644 index 00000000..c87e1acc --- /dev/null +++ b/src/Libs/Middlewares/ExceptionHandlerMiddleware.php @@ -0,0 +1,22 @@ +handle($request); + } catch (\Throwable $e) { + return api_error($e->getMessage(), $e->getCode()); + } + } +}