cphipps/watchstate
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Removed --import-unwatched flag for import, we import the state regardless of watched/unwatched now.

Browse Source
This commit is contained in:
Abdulmhsen B. A. A
2022-04-25 22:27:12 +03:00
parent 7060f30eed
commit 5d95ae5a14
9 changed files with 59 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
FAQ.md
View File

@@ -194,32 +194,16 @@ Flags:
---
### Q: Can Task scheduler import unwatched episodes?
Yes, Set the environment variable `WS_CRON_IMPORT_UNWATCHED` in your `docker-compose.yaml` and restart your container
for changes to take effect.
---
### Q: Can this tool work with alternative Plex agents?
#### Supported Agents GUIDs:
* plex://(type)/(id)
* tvdb://(id)
* imdb://(id)
* tmdb://(id)
* com.plexapp.agents.imdb://(id)?lang=en
* com.plexapp.agents.tmdb://(id)?lang=en
* com.plexapp.agents.themoviedb://(id)?lang=en
* com.plexapp.agents.hama://(db)-(id)
* com.plexapp.agents.xbmcnfo://(id)?lang=en
#### Unsupported Agents:
* com.plexapp.agents.tvdb://(show-id)/(season)/(episode)?lang=en
* com.plexapp.agents.thetvdb://(show-id)/(season)/(episode)?lang=en
* com.plexapp.agents.tvmaze://(show-id)/(season)/(episode)?lang=en
* com.plexapp.agents.xbmcnfotv://(show-id)/(season)/(episode)?lang=en
Those agents do not provide Unique ID, and thus will not work with this tool.
* plex://(type)/(id) `New Plex Agent`
* tvdb://(id) `New Plex Agent`
* imdb://(id) `New Plex Agent`
* tmdb://(id) `New Plex Agent`
* com.plexapp.agents.imdb://(id)?lang=en `(Old plex agents)`
* com.plexapp.agents.tmdb://(id)?lang=en `(Old plex agents)`
* com.plexapp.agents.themoviedb://(id)?lang=en `(Old plex agents)`
* com.plexapp.agents.hama://(db)-(id) `(anime agent parser)`
* com.plexapp.agents.xbmcnfo://(id)?lang=en `( xbmc nfo parser agent)`

31
README.md
View File

@@ -119,8 +119,7 @@ If you don't want to use webhooks and want to rely only on scheduled task for im
of `WS_CRON_IMPORT` to `1`. By default, we run the import command every hour. However, you can change the scheduled task
timer by adding another variable `WS_CRON_IMPORT_AT` and set its value to valid cron expression. for
example, `0 */2 * * *` it will run every two hours instead of 1 hour. beware, this operation is somewhat costly as it's
pulls the entire server library. You can also set `WS_CRON_IMPORT_UNWATCHED` to `1` allow the task to pull unwatched
items.
pulls the entire server library.
---
@@ -304,39 +303,39 @@ None that we are aware of.
- (bool) `WS_REQUEST_DEBUG` enable debug mode for pre webhook request.
- (integer) `WS_WEBHOOK_TOKEN_LENGTH` how many bits for the webhook api key generator.
- (bool) `WS_LOGGER_STDERR_ENABLED` enable stderr output logging.
- (string) `WS_LOGGER_STDERR_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY,
100|200|250|300|400|500|550|600).
- (string) `WS_LOGGER_STDERR_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY).
- (bool) `WS_LOGGER_FILE_ENABLE` enable file logging.
- (string) `WS_LOGGER_FILE_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY,
100|200|250|300|400|500|550|600).
- (string) `WS_LOGGER_FILE_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY).
- (string) `WS_LOGGER_FILE` full path for log file. By default, it's stored at `$(WS_TMP_DIR)/logs/app.log`
- (bool) `WS_LOGGER_SYSLOG_ENABLED` enable syslog logger.
- (int) `WS_LOGGER_SYSLOG_FACILITY` syslog logging facility
- (string) `WS_LOGGER_SYSLOG_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY,
100|200|250|300|400|500|550|600).
- (string) `WS_LOGGER_SYSLOG_LEVEL` level to log (DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY).
- (string) `WS_LOGGER_SYSLOG_NAME` What name should logs be under.
- (int) `WS_CRON_IMPORT` enable import scheduled task.
- (string) `WS_CRON_IMPORT_AT` cron expression timer.
- (bool) `WS_CRON_IMPORT_UNWATCHED` Allow the import task to import unwatched items. Defaults to `false`.
- (string) `WS_CRON_IMPORT_DEBUG_LEVEL` set debug level, to see unmatched items set to `-vv`. Defaults to `-v`
- (int) `WS_CRON_EXPORT` enable export scheduled task.
- (string) `WS_CRON_EXPORT_AT` cron expression timer.
- (string) `WS_CRON_EXPORT_DEBUG_LEVEL` set debug level. Defaults to `-v`
- (int) `WS_CRON_PUSH` enable push scheduled task.
- (string) `WS_CRON_PUSH_AT` cron expression timer.
- (string) `WS_CRON_PUSH_DEBUG_LEVEL` set debug level. Defaults to `-v`
- (int) `WS_CRON_CACHE` enable caching of GUIDs relations.
- (string) `WS_CRON_CACHE_AT` cron expression timer.
- (string) `WS_CRON_CACHE_DEBUG_LEVEL` set debug level. Defaults to `-v`
- (string) `WS_LOGS_PRUNE_AFTER` Delete logs older than specified time, set to `disable` to disable logs pruning. it
follows php [strtotime](https://www.php.net/strtotime) function rules.
- (bool) `WS_DEBUG_IMPORT` Whether to log invalid GUID items from server in `${WS_TMP_DIR}/debug`.
- (bool) `WS_IMPORT_PROMOTE_GUID_ERROR` By default we log this error to `INFO` if this variable set to true it will
promote it to `NOTICE` level.
# Container specific environment variables
- (int) `WS_NO_CHOWN` do not change ownership of `/config` inside container.
- (int) `WS_DISABLE_HTTP` disable included http server.
- (int) `WS_DISABLE_CRON` disable included task scheduler.
- (int) `WS_UID` Container user ID
- (int) `WS_GID` Container group ID
- (int) `WS_NO_CHOWN` do not change ownership needed paths inside container.
- (int) `WS_DISABLE_HTTP` disable included HTTP Server.
- (int) `WS_DISABLE_CRON` disable included Task Scheduler.
- (int) `WS_UID` Container app user ID.
- (int) `WS_GID` Container app group ID.
---
# FAQ

16
config/config.php
View File

@@ -163,7 +163,7 @@ return (function () {
Task::RUN_AT => (string)env('WS_CRON_IMPORT_AT', '0 */1 * * *'),
Task::COMMAND => '@state:import',
Task::ARGS => [
'-v' => null,
env('WS_CRON_IMPORT_DEBUG_LEVEL', '-v') => null,
]
],
ExportCommand::TASK_NAME => [
@@ -172,7 +172,7 @@ return (function () {
Task::RUN_AT => (string)env('WS_CRON_EXPORT_AT', '30 */1 * * *'),
Task::COMMAND => '@state:export',
Task::ARGS => [
'-v' => null,
env('WS_CRON_EXPORT_DEBUG_LEVEL', '-v') => null,
]
],
PushCommand::TASK_NAME => [
@@ -181,7 +181,7 @@ return (function () {
Task::RUN_AT => (string)env('WS_CRON_PUSH_AT', '*/10 * * * *'),
Task::COMMAND => '@state:push',
Task::ARGS => [
'-v' => null,
env('WS_CRON_PUSH_DEBUG_LEVEL', '-v') => null,
]
],
CacheCommand::TASK_NAME => [
@@ -190,13 +190,13 @@ return (function () {
Task::RUN_AT => (string)env('WS_CRON_CACHE_AT', '0 */6 * * *'),
Task::COMMAND => '@state:cache',
Task::ARGS => [
'-v' => null,
env('WS_CRON_CACHE_DEBUG_LEVEL', '-v') => null,
]
],
PruneCommand::TASK_NAME => [
Task::NAME => PruneCommand::TASK_NAME,
Task::ENABLED => (bool)env('WS_CRON_PRUNE', true),
Task::RUN_AT => (string)env('WS_CRON_CACHE_AT', '0 */12 * * *'),
Task::ENABLED => true,
Task::RUN_AT => '0 */12 * * *',
Task::COMMAND => '@config:prune',
Task::ARGS => [
'-v' => null,
@@ -204,9 +204,5 @@ return (function () {
],
];
if (true === (bool)env('WS_CRON_IMPORT_UNWATCHED', false)) {
$config['tasks'][ImportCommand::TASK_NAME][Task::ARGS]['--import-unwatched'] = null;
}
return $config;
})();

16
src/Commands/State/ImportCommand.php
View File

@@ -12,7 +12,6 @@ use App\Libs\Entity\StateInterface;
use App\Libs\Extends\CliLogger;
use App\Libs\Mappers\Import\DirectMapper;
use App\Libs\Mappers\ImportInterface;
use App\Libs\Servers\ServerInterface;
use App\Libs\Storage\PDO\PDOAdapter;
use App\Libs\Storage\StorageInterface;
use Psr\Log\LoggerInterface;
@@ -64,7 +63,7 @@ class ImportCommand extends Command
'timeout',
null,
InputOption::VALUE_REQUIRED,
'Set request timeout in seconds'
'Set request timeout in seconds for each request.'
)
->addOption(
'servers-filter',
@@ -77,7 +76,7 @@ class ImportCommand extends Command
'import-unwatched',
null,
InputOption::VALUE_NONE,
'Import unwatched state (note: It Will set items to unwatched if the server has newer date on items)'
'--DEPRECATED-- will be removed in v1.x. We import the item regardless of watched/unwatched state.'
)
->addOption('stats-show', null, InputOption::VALUE_NONE, 'Show final status.')
->addOption(
@@ -87,7 +86,12 @@ class ImportCommand extends Command
'Filter final status output e.g. (servername.key)',
null
)
->addOption('mapper-direct', null, InputOption::VALUE_NONE, 'Use less memory hungry mapper.')
->addOption(
'mapper-direct',
null,
InputOption::VALUE_NONE,
'Uses less memory. However, it\'s significantly slower then default mapper.'
)
->addOption('config', 'c', InputOption::VALUE_REQUIRED, 'Use Alternative config file.');
}
@@ -194,10 +198,6 @@ class ImportCommand extends Command
$opts = ag($server, 'options', []);
if ($input->getOption('import-unwatched')) {
$opts[ServerInterface::OPT_IMPORT_UNWATCHED] = true;
}
if ($input->getOption('proxy')) {
$opts['client']['proxy'] = $input->getOption('proxy');
}

36
src/Libs/Mappers/Import/DirectMapper.php
View File

@@ -7,7 +7,6 @@ namespace App\Libs\Mappers\Import;
use App\Libs\Data;
use App\Libs\Entity\StateInterface;
use App\Libs\Mappers\ImportInterface;
use App\Libs\Servers\ServerInterface;
use App\Libs\Storage\StorageInterface;
use DateTimeInterface;
use Psr\Log\LoggerInterface;
@@ -21,11 +20,9 @@ final class DirectMapper implements ImportInterface
];
private int $changed = 0;
private string $guidErrorLevel = 'info';
public function __construct(private LoggerInterface $logger, private StorageInterface $storage)
{
$this->guidErrorLevel = true === (bool)env('WS_IMPORT_PROMOTE_GUID_ERROR', false) ? 'notice' : 'info';
}
public function setUp(array $opts): ImportInterface
@@ -41,7 +38,7 @@ final class DirectMapper implements ImportInterface
public function add(string $bucket, string $name, StateInterface $entity, array $opts = []): self
{
if (!$entity->hasGuids()) {
$this->logger->{$this->guidErrorLevel}(sprintf('Ignoring %s. No valid GUIDs.', $name));
$this->logger->info(sprintf('Ignoring %s. No valid GUIDs.', $name));
Data::increment($bucket, $entity->type . '_failed_no_guid');
return $this;
}
@@ -49,12 +46,6 @@ final class DirectMapper implements ImportInterface
$item = $this->get($entity);
if (null === $entity->id && null === $item) {
if (0 === $entity->watched && true !== ($opts[ServerInterface::OPT_IMPORT_UNWATCHED] ?? false)) {
$this->logger->debug(sprintf('Ignoring %s. Not watched.', $name));
Data::increment($bucket, $entity->type . '_ignored_not_watched');
return $this;
}
try {
$this->storage->insert($entity);
} catch (Throwable $e) {
@@ -70,31 +61,6 @@ final class DirectMapper implements ImportInterface
return $this;
}
// -- Ignore unwatched Item.
if (0 === $entity->watched && true !== ($opts[ServerInterface::OPT_IMPORT_UNWATCHED] ?? false)) {
// -- check for updated GUIDs.
if ($item->apply($entity, guidOnly: true)->isChanged()) {
try {
$this->changed++;
if (!empty($entity->meta)) {
$item->meta = $entity->meta;
}
$this->storage->update($item);
$this->operations[$entity->type]['updated']++;
$this->logger->debug(sprintf('Updating %s. GUIDs.', $name), $item->diff());
return $this;
} catch (Throwable $e) {
$this->operations[$entity->type]['failed']++;
Data::append($bucket, 'storage_error', $e->getMessage());
return $this;
}
}
$this->logger->debug(sprintf('Ignoring %s. Not watched.', $name));
Data::increment($bucket, $entity->type . '_ignored_not_watched');
return $this;
}
// -- Ignore old item.
if (null !== ($opts['after'] ?? null) && ($opts['after'] instanceof DateTimeInterface)) {
if ($opts['after']->getTimestamp() >= $entity->updated) {

31
src/Libs/Mappers/Import/MemoryMapper.php
View File

@@ -7,7 +7,6 @@ namespace App\Libs\Mappers\Import;
use App\Libs\Data;
use App\Libs\Entity\StateInterface;
use App\Libs\Mappers\ImportInterface;
use App\Libs\Servers\ServerInterface;
use App\Libs\Storage\StorageInterface;
use DateTimeInterface;
use Psr\Log\LoggerInterface;
@@ -33,11 +32,8 @@ final class MemoryMapper implements ImportInterface
private bool $fullyLoaded = false;
private string $guidErrorLevel = 'info';
public function __construct(private LoggerInterface $logger, private StorageInterface $storage)
{
$this->guidErrorLevel = true === (bool)env('WS_IMPORT_PROMOTE_GUID_ERROR', false) ? 'notice' : 'info';
}
public function setUp(array $opts): ImportInterface
@@ -65,18 +61,12 @@ final class MemoryMapper implements ImportInterface
public function add(string $bucket, string $name, StateInterface $entity, array $opts = []): self
{
if (!$entity->hasGuids()) {
$this->logger->{$this->guidErrorLevel}(sprintf('Ignoring %s. No valid GUIDs.', $name));
$this->logger->info(sprintf('Ignoring %s. No valid GUIDs.', $name));
Data::increment($bucket, $entity->type . '_failed_no_guid');
return $this;
}
if (false === ($pointer = $this->getPointer($entity))) {
if (0 === $entity->watched && true !== ($opts[ServerInterface::OPT_IMPORT_UNWATCHED] ?? false)) {
$this->logger->debug(sprintf('Ignoring %s. Not watched.', $name));
Data::increment($bucket, $entity->type . '_ignored_not_watched');
return $this;
}
$this->objects[] = $entity;
$pointer = array_key_last($this->objects);
@@ -89,25 +79,6 @@ final class MemoryMapper implements ImportInterface
return $this;
}
// -- Ignore unwatched Item.
if (0 === $entity->watched && true !== ($opts[ServerInterface::OPT_IMPORT_UNWATCHED] ?? false)) {
// -- check for updated GUIDs.
if ($this->objects[$pointer]->apply($entity, guidOnly: true)->isChanged()) {
$this->changed[$pointer] = $pointer;
if (!empty($entity->meta)) {
$this->objects[$pointer]->meta = $entity->meta;
}
Data::increment($bucket, $entity->type . '_updated');
$this->addGuids($this->objects[$pointer], $pointer);
$this->logger->debug(sprintf('Updating %s. GUIDs.', $name), $this->objects[$pointer]->diff());
return $this;
}
$this->logger->debug(sprintf('Ignoring %s. Not watched.', $name));
Data::increment($bucket, $entity->type . '_ignored_not_watched');
return $this;
}
// -- Ignore old item.
if (null !== ($opts['after'] ?? null) && ($opts['after'] instanceof DateTimeInterface)) {
if ($opts['after']->getTimestamp() >= $entity->updated) {

8
src/Libs/Servers/JellyfinServer.php
View File

@@ -74,14 +74,11 @@ class JellyfinServer implements ServerInterface
protected array $cacheData = [];
protected string|int|null $uuid = null;
protected string $guidErrorLevel = 'info';
public function __construct(
protected HttpClientInterface $http,
protected LoggerInterface $logger,
protected CacheInterface $cache
) {
$this->guidErrorLevel = true === (bool)env('WS_IMPORT_PROMOTE_GUID_ERROR', false) ? 'notice' : 'info';
}
/**
@@ -1296,7 +1293,7 @@ class JellyfinServer implements ServerInterface
$guids = (array)($item->ProviderIds ?? []);
$this->logger->{$this->guidErrorLevel}(
$this->logger->info(
sprintf(
'Ignoring %s. No valid GUIDs. Possibly unmatched item?',
$iName
@@ -1355,8 +1352,7 @@ class JellyfinServer implements ServerInterface
];
$mapper->add($this->name, $iName, Container::get(StateInterface::class)::fromArray($row), [
'after' => $after,
self::OPT_IMPORT_UNWATCHED => (bool)($this->options[self::OPT_IMPORT_UNWATCHED] ?? false),
'after' => $after
]);
} catch (Throwable $e) {
$this->logger->error($e->getMessage(), [

43
src/Libs/Servers/PlexServer.php
View File

@@ -45,6 +45,15 @@ class PlexServer implements ServerInterface
'anidb' => Guid::GUID_ANIDB,
];
protected const DISABLED_GUID_AGENTS = [
'local',
'com.plexapp.agents.none',
'com.plexapp.agents.tvdb',
'com.plexapp.agents.thetvdb',
'com.plexapp.agents.tvmaze',
'com.plexapp.agents.xbmcnfotv',
];
protected const WEBHOOK_ALLOWED_TYPES = [
'movie',
'episode',
@@ -78,14 +87,11 @@ class PlexServer implements ServerInterface
protected string|int|null $uuid = null;
protected string|int|null $user = null;
protected string $guidErrorLevel = 'info';
public function __construct(
protected HttpClientInterface $http,
protected LoggerInterface $logger,
protected CacheInterface $cache
) {
$this->guidErrorLevel = true === (bool)env('WS_IMPORT_PROMOTE_GUID_ERROR', false) ? 'notice' : 'info';
}
/**
@@ -1336,12 +1342,13 @@ class PlexServer implements ServerInterface
}
}
$this->logger->{$this->guidErrorLevel}(
$this->logger->info(
sprintf('Ignoring %s. No valid GUIDs. Possibly unmatched item?', $iName),
[
'guids' => empty($item->Guid) ? 'None' : $item->Guid,
]
);
Data::increment($this->name, $type . '_ignored_no_supported_guid');
return;
}
@@ -1389,7 +1396,6 @@ class PlexServer implements ServerInterface
$mapper->add($this->name, $iName, Container::get(StateInterface::class)::fromArray($row), [
'after' => $after,
self::OPT_IMPORT_UNWATCHED => (bool)($this->options[self::OPT_IMPORT_UNWATCHED] ?? false),
]);
} catch (Throwable $e) {
$this->logger->error($e->getMessage(), [
@@ -1513,37 +1519,22 @@ class PlexServer implements ServerInterface
* com.plexapp.agents.hama://(db)-(id)
* com.plexapp.agents.xbmcnfo://(id)?lang=xn > imdb
* @Disabled For:
* local://(id)
* com.plexapp.agents.none://(gid)?lang=en
* com.plexapp.agents.tvdb://(show-id)/(season)/(episode)?lang=en
* com.plexapp.agents.thetvdb://(show-id)/(season)/(episode)?lang=en
* com.plexapp.agents.tvmaze://(show-id)/(season)/(episode))?lang=en
* com.plexapp.agents.xbmcnfotv://(show-id)/(season)/(episode)?lang=xn
*/
$this->logger->debug('Parsing Legacy plex content agent.', ['guid' => $agent]);
$disabled = [
'com.plexapp.agents.tvdb',
'com.plexapp.agents.thetvdb',
'com.plexapp.agents.tvmaze',
'com.plexapp.agents.xbmcnfotv',
];
if (in_array(before($agent, '://'), $disabled)) {
$this->logger->{$this->guidErrorLevel}(
'Unable to parse GUID as it does not provide episode unique id',
['guid' => $agent]
);
if (true === in_array(before($agent, '://'), self::DISABLED_GUID_AGENTS)) {
return $agent;
}
try {
if (str_starts_with($agent, 'com.plexapp.agents.none')) {
return $agent;
}
$replacer = [
'agents.themoviedb' => 'agents.tmdb',
'agents.xbmcnfo://' => 'agents.imdb://',
'com.plexapp.agents.themoviedb://' => 'com.plexapp.agents.tmdb://',
'com.plexapp.agents.xbmcnfo://' => 'com.plexapp.agents.imdb://',
];
$agent = str_replace(array_keys($replacer), array_values($replacer), $agent);
@@ -1554,7 +1545,7 @@ class PlexServer implements ServerInterface
}
}
$id = afterlast($agent, 'agents.');
$id = afterLast($agent, 'agents.');
$agentGuid = explode('://', $id);
$agent = $agentGuid[0];
$guid = explode('/', $agentGuid[1])[0];

1
src/Libs/Servers/ServerInterface.php
View File

@@ -17,7 +17,6 @@ use Symfony\Contracts\HttpClient\ResponseInterface;
interface ServerInterface
{
public const OPT_IMPORT_UNWATCHED = 'importUnwatched';
public const OPT_EXPORT_IGNORE_DATE = 'exportIgnoreDate';
/**