cphipps/watchstate
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Support loading environment variables in the docker entrypoint using .env file.

Browse Source
This commit is contained in:
abdulmohsen
2022-07-20 19:04:06 +03:00
parent 110831d091
commit 166b1f8f83
5 changed files with 5132 additions and 5249 deletions
Download Patch File Download Diff File Expand all files Collapse all files

284
FAQ.md
View File

@@ -1,20 +1,21 @@
# FAQ
**All commands have help document attached to them. So, please read them.**
To see list of all the available commands run
### Q: How to sync play state to a new backend without overwriting current play state?
Add the backend and when asked, answer `no` for `Enable Importing metadata and play state from this backend?`. when you
finish, then run the following command:
```bash
$ docker exec -ti watchstate console state:export -vvfis [BACKEND_NAME]
```shell
$ docker exec -ti watchstate console list
```
this command will force export your current local play state to the selected backend. If the operation is
successful you can then enable the full import from the backend.
This command will list all available commands, and each command has help document attached to it, simply run
---
```shell
$ docker exec -ti watchstate console help [COMMAND_NAME]
```
It will show you the relevant information regarding the command and some frequently asked question about that command.
The help document attach to each command is more up to date and precise. So, please read it.
----
### Q: Is there support for Multi-user setup?
@@ -22,9 +23,7 @@ No, The tool is designed to work for single user. However, It's possible to run
use single container for all users, however it's not really easy refer
to [issue #136](https://github.com/ArabCoders/watchstate/issues/136).
#### Note
Note: for Plex home/managed users run the following command to extract each managed user access token.
**Note**: for Plex home/managed users run the following command to extract each managed user access token.
```bash
$ docker exec -ti console backend:users:list --with-tokens -- [BACKEND_NAME]
@@ -34,26 +33,6 @@ For Jellyfin/Emby, you can just generate new API tokens.
----
### Q: How to get library id?
Run the following command to get list of backend libraries.
```bash
$ docker exec -ti watchstate console backend:library:list [BACKEND_NAME]
```
it should display something like
| Id | Title | Type | Ignored | Supported |
|-----|-------------|--------|---------|-----------|
| 1 | Movies | Movie | No | Yes |
| 2 | TV Shows | Show | No | Yes |
| 3 | Audio Books | Artist | Yes | No |
The id column refers to backend side `library id`.
---
### Q: Can this tool run without docker?
Yes, if you have the required PHP version and the needed extensions. to run this tool you need the following `php8.1`,
@@ -79,72 +58,12 @@ if you want a webhook support you would need a frontend proxy for `php8.1-fpm` l
---
### Q: Some records keep getting updated even when the state did not change?
This most likely means your media backends have conflicting external ids for the reported items, and thus triggering an
update as the tool see different external ids on each event from the backends. In our testing we noticed that at least
few hundred records in thetvdb that get reported by plex have incorrect imdb external id, which in turns conflicts
sometimes with jellyfin/emby there is nothing we can do beside have the problematic records reported to thetvdb site
mods to fix their db entries.
----
### Q: What does "No valid/supported external ids." in logs means ?
This most likely means that the item is not matched in your media backend
* jellyfin/emby: Edit metadata and make sure there are external ids listed in the metadata.
* For plex click the (...), and click Fix match. after that refresh metadata.
For episodes, we support both external ids like movies and relative unique ids, To make episodes sync work at least one
of the following conditions has to be true:
* The episode should have external ids.
* The parent show should have external ids, to make relative unique id works.
---
### Q: Does this tool require webhooks to work?
No, You can use the task scheduler or on demand sync if you want.
---
### Q: How to see my data?
```bash
$ docker exec -ti watchstate console db:list
```
This command will give you access to see the database entries. by default, it will show the last 20 items, however you
can run the same command with `[-h, --help]` to see more options to extend the list or to filter the results.
---
### Q: How to ignore specific libraries from being processed?
Run the following command:
```bash
$ docker exec -ti watchstate console config:edit --key options.ignore --set 'id1,id2,id3' -- [BACKEND_NAME]
```
where `id1,id2,id3` refers to backend library id
If you ignored a library by mistake you can run the same command again and omit the id, or you can just delete the key
entirely by running the following command
```bash
$ docker exec -ti watchstate console config:edit --delete --key options.ignore -- [BACKEND_NAME]
```
##### Note
While this feature works for manual/task scheduler for all supported backends, jellyfin/emby does not report library id
via webhook events. So, this feature will not work for them in webhook context and the items will be processed.
---
### Q: I get tired of writing the whole command everytime is there an easy way run the commands?
Since there is no way to access the command interface outside docker, you can create small shell script to at least omit
@@ -184,80 +103,7 @@ where `600` is the number of secs before the timeout handler will kill the reque
---
### Q: How to perform search on backend libraries?
Use the following command:
```bash
$ docker exec -ti console backend:search:query [BACKEND_NAME] '[QUERY_STRING]'
```
where `[QUERY_STRING]` is the keyword that you want to search for
### Optional flags
* `[-l, --limit]` To limit returned results. Default to `25`.
* `[-o, --output]` Set output style, it can be `yaml`, `json` or `table`. Default to `table`.
* `[--include-raw-response]` will include backend response in main response body with `raw` key.
---
### Q: How to get metadata about specific item id?
Use the following command:
```bash
$ docker exec -ti watchstate console backend:search:id [BACKEND_NAME] [BACKEND_ITEM_ID]
```
where `[BACKEND_ITEM_ID]` refers to backend item id
### Optional flags
* `[-o, --output]` Set output style, it can be `yaml`, `json` or `table`. Default to `table`.
* `[--include-raw-response]` will include backend response in main response body with `raw` key.
---
### Q: How to look for mis-identified items?
Use the `backend:library:mismatch` command. For example,
```bash
$ docker exec -ti watchstate console backend:library:mismatch [BACKEND_NAME] [LIBRARY_ID]
```
where `[LIBRARY_ID]` refers to backend library id
### Optional flags
* `[-p, --percentage]` How much in percentage the title has to be in path to be marked as matched item. Default
to `50.0%`.
* `[-o, --output]` Set output mode, it can be `yaml`, `json` or `table`. Default to `table`.
* `[-m, --method]` Which method to use, it can be `similarity`, or `levenshtein`. Default to `similarity`.
* `[--include-raw-response]` Will include backend response in main response body with `raw` key.
---
### Q: How to look for unmatched items?
Use the `backend:library:unmatched` command. For example,
```bash
$ docker exec -ti watchstate console backend:library:unmatched [BACKEND_NAME] [LIBRARY_ID]
```
where `[LIBRARY_ID]` refers to backend library id
### Optional flags
* `[-o, --output]` Set output mode, it can be `yaml`, `json` or `table`. Defaults to `table`.
* `[--show-all]` Will show all library items regardless of the match status.
* `[--include-raw-response]` Will include backend response in main response body with `raw` key.
---
### Q: Which external ids supported for Plex?
### Q: Which external db ids supported for Plex Media Server?
* tvdb://(id) `New Plex Agent`
* imdb://(id) `New Plex Agent`
@@ -272,7 +118,7 @@ where `[LIBRARY_ID]` refers to backend library id
---
### Q: Which external ids supported for Jellyfin/Emby?
### Q: Which external db ids supported for Jellyfin and Emby?
* imdb://(id)
* tvdb://(id)
@@ -283,33 +129,22 @@ where `[LIBRARY_ID]` refers to backend library id
---
### Q: What does mapper mean?
### Environment Variables
A Mapper is class that have list of all external ids that point to record in database. think of it as dictionary that
reference to specific item.
There are many ways to load the environment variables, However the recommended methods are:
#### Memory Mapper (Default)
* Via `docker-compose.yaml` file.
* Via `/config/config/.env` file. This file normally does not exist you have to created manually.
Memory mapper is the Default mapper, it uses memory to load the entire state table into memory, which in turn leads
to better performance. you shouldn't use the other mapper unless you are running into memory problems.
to see list of loaded environment variables run:
#### Direct Mapper
```shell
$ docker exec -ti watchstate console system:env
```
Direct mapper is suitable for more memory constraint systems, it just loads the external ids mapping into memory,
however it does not keep the state in memory, thus uses less memory compared to `MemoryMapper`, But the trade-off is
it's slower than `MemoryMapper`.
#### Tool specific environment variables.
#### Comparison between mappers
| Operation | Memory Mapper | Direct Mapper |
|----------------|---------------|-----------------|
| Memory Usage | (✗) Higher | (✓) Lower |
| Matching Speed | (✓) Faster | (✗) Slower |
| DB Operations | (✓) Faster | (✗) Slower |
---
### Q: What environment variables supported?
These environment variables relates to the tool itself, you can load them via the recommended methods.
| Key | Type | Description | Default |
|--------------------------|--------|-------------------------------------------------------------------------------|--------------------|
@@ -325,7 +160,7 @@ it's slower than `MemoryMapper`.
| WS_WEBHOOK_DEBUG | bool | If enabled, allow dumping request/webhook using `rdump` & `wdump` parameters. | `false` |
| WS_EPISODES_DISABLE_GUID | bool | Disable external id parsing for episodes and rely on relative ids. | `true` |
Note for environment variables that has `{TASK}` you should replace it with one
**Note**: for environment variables that has `{TASK}` tag, you **MUST** replace it with one
of `IMPORT`, `EXPORT`, `PUSH`, `BACKUP`, `PRUNE`, `INDEXES`. To see tasks active settings run
```bash
@@ -334,14 +169,17 @@ $ docker exec -ti watchstate console system:tasks
#### Container specific environment variables.
| Key | Type | Description | Default |
|------------------|---------|----------------------------------------------------------------------|---------|
| WS_DISABLE_CHOWN | integer | Do not change ownership for needed directories inside the container. | `0` |
| WS_DISABLE_HTTP | integer | Disable included HTTP Server. | `0` |
| WS_DISABLE_CRON | integer | Disable included Task Scheduler. | `0` |
| WS_DISABLE_CACHE | integer | Disable included Cache Server. | `0` |
| WS_UID | integer | Set container user id. | `1000` |
| WS_GID | integer | Set container group id. | `1000` |
These environment variables relates to the container itself, and it's recommended to load them
via the `docker-compose.yaml` file.
| Key | Type | Description | Default |
|------------------|---------|----------------------------------------------|---------|
| WS_DISABLE_CHOWN | integer | Do not change ownership `/config` directory. | `0` |
| WS_DISABLE_HTTP | integer | Disable included `HTTP Server`. | `0` |
| WS_DISABLE_CRON | integer | Disable included `Task Scheduler`. | `0` |
| WS_DISABLE_CACHE | integer | Disable included `Cache Server`. | `0` |
| WS_UID | integer | Set container user id. | `1000` |
| WS_GID | integer | Set container group id. | `1000` |
---
@@ -356,11 +194,11 @@ Via reverse proxy : `https://watchstate.domain.example/?apikey=[WEBHOOK_TOKEN]`.
If your media backend support sending headers then remove query parameter `?apikey=[WEBHOOK_TOKEN]`, and add this header
```http request
```
X-apikey: [WEBHOOK_TOKEN]
```
where `[WEBHOOK_TOKEN]` Should match the backend `webhook.token` value. To see your backends webhook tokens run:
where `[WEBHOOK_TOKEN]` Should match the backend `webhook.token` value. To see your webhook token for each backend run:
```bash
$ docker exec -ti watchstate console config:view webhook.token
@@ -372,7 +210,9 @@ If you see 'Not configured, or invalid key.' or empty value. run the following c
$ docker exec -ti watchstate console config:edit --regenerate-webhook-token -- [BACKEND_NAME]
```
#### Emby (you need "Emby Premiere" to use webhooks).
-----
#### Emby (you need `Emby Premiere` to use webhooks).
Go to your Manage Emby Server > Server > Webhooks > (Click Add Webhook)
@@ -380,18 +220,22 @@ Go to your Manage Emby Server > Server > Webhooks > (Click Add Webhook)
`http://localhost:8081/?apikey=[WEBHOOK_TOKEN]`
##### Webhook Events
Select the following events
##### Webhook Events:
* Playback events
* User events
##### Limit user events to:
* Select your user.
Click `Add Webhook`
#### Plex (you need "PlexPass" to use webhooks)
-----
Go to your plex Web UI > Settings > Your Account > Webhooks > (Click ADD WEBHOOK)
#### Plex (You need `Plex Pass` to use webhooks)
Go to your Plex Web UI > Settings > Your Account > Webhooks > (Click ADD WEBHOOK)
##### URL:
@@ -399,9 +243,8 @@ Go to your plex Web UI > Settings > Your Account > Webhooks > (Click ADD WEBHOOK
Click `Save Changes`
#### Note:
If you have multiple plex backends and use the same PlexPass account for all of them, you have to unify the API key, by
**Note:** If you use multiple plex servers and use the same PlexPass account for all of them, you have to unify the API
key, by
running the following command:
```bash
@@ -410,13 +253,21 @@ Plex global webhook API key is: [random_string]
```
The reason is due to the way plex handle webhooks, And to know which webhook request belong to which backend we have to
identify the backends, The unify command will do the necessary adjustments to handle multi plex backend setup. for more
identify the backends, The unify command will do the necessary adjustments to handle multiple plex servers setup. for
more
information run.
```bash
$ docker exec -ti watchstate console help config:unify
```
**Note**: If you share your plex server with other users, i,e. `Home/managed users`, you have to enable match user id,
otherwise
their play state will end up changing your play state. Plex will still send their events. But with match user id they
will be ignored.
-----
#### Jellyfin (Free)
go to your jellyfin dashboard > plugins > Catalog > install: Notifications > Webhook, restart your jellyfin. After that
@@ -424,7 +275,7 @@ go back again to dashboard > plugins > webhook. Add `Add Generic Destination`,
##### Webhook Name:
Choose whatever name you want. For example, `Watchstate-Webhook`
`Watchstate-Webhook`
##### Webhook Url:
@@ -432,8 +283,6 @@ Choose whatever name you want. For example, `Watchstate-Webhook`
##### Notification Type:
Select the following events
* Item Added
* User Data Saved
* Playback Start
@@ -479,12 +328,10 @@ Those are some Webhook limitations we discovered for the following media backend
items. [See feature request](https://emby.media/community/index.php?/topic/97889-new-content-notification-webhook/).
* Emby webhook test event does not contain data. To test if your setup works, play something or do mark an item as
played or unplayed you should see changes reflected in `docker exec -ti watchstate console db:list`.
* Items might be marked as unplayed if Libraries > Advanced - `Date added behavior for new content:` is set
to `Use date scanned into library`. This happens if the media file has been replaced.
#### Jellyfin
* If you don't select a user id, the Plugin will send `itemAdd` event without user data, and will fail the check if
* If you don't select a user id, the plugin will send `itemAdd` event without user data, and will fail the check if
you happen to enable `webhook.match.user` for jellyfin.
* Sometimes jellyfin will fire webhook `itemAdd` event without the item being matched.
* Even if you select user id, sometimes `itemAdd` event will fire without user data.
@@ -497,4 +344,5 @@ Those are some Webhook limitations we discovered for the following media backend
As stated in webhook limitation section sometimes media backends don't make it easy to receive those events, as such, to
complement webhooks, you should enable import/export tasks by settings their respective environment variables in
your `docker-compose.yaml` file.
your `docker-compose.yaml` file. For more information run help on `system:env` command as well as `system:tasks`
command.

21
README.md
View File

@@ -1,9 +1,18 @@
# WatchState
WatchState is a tool that can sync your play state cross your media backends, without relying on 3rd party services,
this tool support `Jellyfin`, `Plex Media Server` and `Emby`. It's also come with some features, like
finding `mis-identified` or `unmatched` items, and the ability to
`search` your backend for specific `item id` or `title`.
This tool primary goal is to sync your backends play state without relying on third party services,
out of the box, this tool support `Jellyfin`, `Plex` and `Emby` media servers.
# Features
* Sync backends play state (from many to many).
* Backup your backends play state into `portable` format.
* Webhook play state receiver.
* Find `un-matched` media items.
* FInd `mis-matched` media items.
* Search your backend for `title`.
* Search your backend for `item id`.
* Display and filter your play state can be exported to `json` or `yaml`.
# Install
@@ -68,7 +77,7 @@ $ docker exec -ti watchstate console state:import -v --select-backends 'home_ple
Now that you have imported your current play state enable the import task by adding the following environment variables
to your `docker-compose.yaml` file `WS_CRON_IMPORT=1`. By default, we have it disabled. for more environment variables
please refer to [Environment variables list](FAQ.md#q-what-environment-variables-supported).
please refer to [Environment variables list](FAQ.md#environment-variables).
### Supported import methods
@@ -101,7 +110,7 @@ $ docker exec -ti watchstate console state:export -v --select-backends 'home_ple
Now that you have exported your current play state, enable the export task by adding the following environment variables
to your `docker-compose.yaml` file `WS_CRON_EXPORT=1`. By default, we have it disabled. for more environment variables
please refer to [Environment variables list](FAQ.md#q-what-environment-variables-supported).
please refer to [Environment variables list](FAQ.md#environment-variables).
---

8
composer.json
View File

@@ -16,10 +16,11 @@
"require": {
"php": ">=8.1",
"ext-pdo": "*",
"ext-pdo_sqlite": "*",
"ext-mbstring": "*",
"ext-ctype": "*",
"ext-curl": "*",
"ext-sqlite3": "*",
"ext-sodium": "*",
"monolog/monolog": "^2.3",
"symfony/console": "^6.0",
"symfony/cache": "^6.0",
@@ -38,7 +39,10 @@
"halaxa/json-machine": "^1.0"
},
"suggest": {
"ext-redis": "For caching"
"ext-sockets": "For UDP commincations.",
"ext-posix": "to check if running under super user.",
"ext-redis": "For caching.",
"ext-simplexml": "For backends that send xml responses."
},
"require-dev": {
"roave/security-advisories": "dev-latest",

10048
composer.lock generated
View File

File diff suppressed because it is too large Load Diff

20
docker/files/entrypoint.sh
View File

@@ -1,6 +1,22 @@
#!/usr/bin/env sh
#!/usr/bin/env bash
set -e
ENV_FILE="${WS_DATA_PATH:-/config}/config/.env"
TIME_DATE=$(date +"%Y-%m-%dT%H:%M:%S%z")
if [ -f "${ENV_FILE}" ]; then
echo "[${TIME_DATE}] INFO: Loading environment variables from [${ENV_FILE}]."
while read -r LINE; do
if [[ $LINE == *'='* ]] && [[ $LINE != '#'* ]]; then
ENV_VAR="$(echo "${LINE}" | envsubst)"
eval "declare -x $ENV_VAR"
fi
done <"${ENV_FILE}"
else
echo "[${TIME_DATE}] INFO: No environment file present at [${ENV_FILE}]."
fi
WS_UID=${WS_UID:-1000}
WS_GID=${WS_GID:-1000}
WS_DISABLE_CHOWN=${WS_DISABLE_CHOWN:-0}
@@ -36,7 +52,7 @@ if [ ! -f "/usr/bin/run-app-cron" ]; then
fi
if [ 0 = "${WS_DISABLE_CHOWN}" ]; then
if ! cat /proc/mounts | grep '/app'; then
if ! grep '/app' /proc/mounts; then
chown -R www-data:users /app
fi
chown -R www-data:users /config /var/lib/nginx/ /etc/redis.conf