As seen in https://groups.google.com/forum/#!msg/octoprint/DyXdqhR0U7c/kKMUsMmIBgAJ
a broken entry_points.txt in some arbitrary Python package installed in
the same python envrionment as OctoPrint can make our whole plugin
detection fail and hence interrupt regular server startup.
This adds better protection against such cases.
A centralized plugin blacklist read from plugins.octoprint.org on server
start now allows us to stop plugins from being loaded that are known to
cause severe issues with OctoPrint's regular operation.
Blacklist entries can be restricted for specific plugin versions &
OctoPrint versions, allowing for very granular control of any kind of
blocking.
Additionally users may disable blacklist processing in general (an
opt-in wizard and a new section in the settings have been added) and
at server start via the new --ignore-blacklist parameter available for
"octoprint serve" and "octoprint daemon".
If a plugin is blacklisted, OctoPrint will not even import the
plugin module in question (if only a plugin key is specified OR a key
and a version and the plugin's version is already known during import
time, which is the case for plugins loaded from entry points) or at the
very least stop the plugin from being enabled (if a plugin key and a
version is specified and the plugin's version is only known after
loading, which is the case for plugins loaded from directories).
* Don't restrict the list of compatibility values to check against
to only those we have mapped, also support unmapped more exotic
ones.
* Allow 1:1 check against sys.platform values (with startswith).
Combined with the above that allows very granular compatibility
modelling ("freebsd11", "freebsd12") if required.
* Instead of only whitelisting ("linux", "freebsd") now also black
listing is possible ("!windows").
A detected os must match all provided whitelist elements (if the
whitelist is empty that is considered always the case) and none of
the backlist elements (if the blacklist is empty that is also
considered always the case).
See the included unit tests for examples of how this works.
* Extended plugin metadata by new property, only evaluated for bundled
plugins.
* Adjusted plugin manager to evaluate new metadata and add
confirmation dialog with details when attempting to disable such
a plugin.
Can be enabled either through new --safe command line
parameter or through server.startOnceInSafeMode in
config.yaml
When running in safe mode the plugin manager will
only allow to disable or uninstall third party plugins. Enabling
third party plugins or installing new plugins is disabled.
That will hopefully allow for more straightforward recovery
in case of a misbehaving plugin.
Most caching is left to the client, by utilizing ETag and Last-Modified headers.
Where it was easily achievable, an additional server side miniature cache of intermediary
results was introduced (e.g. for the files). The regular cached decorator was not used
since it targets caching full responses, and the responses in question already contained
client request specific data. Caching "one step earlier" allows better usage of the cache here.
Also introduced a dependency on the scandir module, to get a bit of a performance boost
on os.walk and os.listdir (which have been replaced with scandir.walk and scandir.listdir
respectively). See https://github.com/benhoyt/scandir#background on why that made
sense.
In case the user site packages are not yet part of the used
working set OR the sys path and ENABLE_USER_SITE is true, the
manager will now make sure that the folder is searched for plugins
as well upon plugin reload.
This is necessary since Python will not automatically include the
user site directory upon firing up the program in case there's
nothing installed to it/it doesn't exist. If a plugin is installed
during run time with --user that will lead to it not being found,
which is undesirable. Hence run time manipulation of sys.path and
the workingset becomes necessary.
Managable currently pretty much only means "uninstallable".
Plugins are managable if their installation location is writable
and - if they are installed from an entry point and OctoPrint is
running in a virtual environment - within the bounds of the virtual
environment (because otherwise pip will not allow to uninstall).
Custom exception should be derived from Exception, not BaseException.
Not only is this specified in the python documentation, but also
tornado will be able to handle Exceptions in requests perfectly fine
and return an HTTP 500 for them, but crash hard (as in, server shut
down follows) for BaseExceptions.
(cherry picked from commit 29b047b)
Custom exception should be derived from Exception, not BaseException.
Not only is this specified in the python documentation, but also
tornado will be able to handle Exceptions in requests perfectly fine
and return an HTTP 500 for them, but crash hard (as in, server shut
down follows) for BaseExceptions.
Sometimes they will still get discovered by python even though their packages have since been uninstalled.
This will also lead to them being reloaded after an uninstall and a subsequent plugin reload.
Marking them as uninstalled and not handing out uninstalled plugins when collecting them solves this.
Since it cannot be reliably determined by the system if a hook is essential for a plugin's functionality or not, it makes more sense to just disable plugins that utilize obsolete hooks then risk running half working plugins.
The new hook allows extending the list of rules for maximum body sizes differing from the default of 100KB and can be used by plugins to allow uploads to them that exceed that file size.
Also extended the plugin manager to detect plugins that implement restart needing hooks (such as the above one) and handling those plugins the same as plugins containing implementations that inherit from octoprint.plugin.core.RestartNeedingPlugin
Plugins may be enabled and disabled during runtime. If they are of types which allow hot loading, this will be done. Otherwise they will be marked as pending and updated after a restart. Same for installation and uninstallation.
