Commit Graph

37 Commits

Author SHA1 Message Date
Nikolay Starodubtsev
975bcc4a98 Log refactoring close to new logging spec
* correct log levels
* correct log messages

partial-bug: 1419872
partial-implements-bp: unified-style-logging

Change-Id: I80c2b4c809dc52125f9f4a3beec0cd3b6c4c1213
2015-09-09 20:54:26 +03:00
Filip Blaha
3cef984de9 Environment modify actions introduced
Base class for modify actions.
Entry points based action registration.
Action manager loads and execute actions.
Out-of-the-box actions implementation.

Partially implements: blueprint policy-based-env-modification

Change-Id: Ib148c5f2b7efb1186d77f5461ef9324cca1287c2
2015-09-09 12:47:45 +02:00
Stan Lagun
16030aeec8 yaql context versioning
This commit reworks mechanism how context chain
is built at each point. The goal was to make
entire chain be dependent on format version
number. This would allow for example to use
yaql 1.0 legacy mode context for MuranoPL/1.0
and not to use it for 2.0.

Not the entire chain is built on demand up to the
root instead of attaching to existed parent context
as it was before. DSL host (engine) is no more
required to have custom MuranoDslExecutor implementation
but a new ContextManager interface that is used
to control contexts on each layer. Engine uses
this interface to register system classes and
engine level yaql functions. Also a ContextManager
is a foundation for method mocking because now
all MuranoPL methods are yaql functions stored
in context and with ContextManager it is possible
to inject/replace function in each scope (
global, package, class, object)

Partially implements: blueprint murano-versioning

Change-Id: I0a553e8044061fe780a83bc04d70b8f80580988f
2015-09-04 14:37:14 +00:00
Stan Lagun
615a4469b3 Restores back plugin support
During reworking of how MuranoPL works with packages and
removal or class-loader plugin support code was broken and
temporary disabled. This commit restores previous functionality
using new PackageLoader and MuranoPackage classes

Change-Id: Id43807bfbb9093ddf9a1fbd9273d2fc34e8e5f59
2015-09-03 12:07:18 +00:00
Stan Lagun
068831ccd8 Package versioning
With this change MuranoPackage becomes first-class DSL citizen.
Packages have version, runtime_version (that is specified
in Format attribute of the manifest file) and a list of classes.
Previously engine used to have package loader which had most
of "load" functionality and class loader that mostly acted as an
adapter from package loader to interface that DSL used to
get classes. Now class loader is gone and is replaced with
package loader at the DSL level. Package loader is responsible
for loading packages by either package or class name (as it was
before) plus semantic_version spec (for example ">=1.2,<2.0").
Package loader can now keep track of several versions of the same
package.

Also packages now have requirements with version specs.
All class names that are encountered in application code are
looked up within requirements only. As a consequence
packages that use other packages without referencing
them explicitly will become broken. An exception from this rule
is core library which is referenced automatically.

Partially implements: blueprint murano-versioning

Change-Id: I8789ba45b6210e71bf4977a766f82b66d2a2d270
2015-09-03 12:06:42 +00:00
Stan Lagun
75afcd238b Fixes Congress model validation
Change-Id: I8293996b936503b10f67ce428c0a17dc059e6481
2015-08-27 16:21:51 +02:00
Stan Lagun
425766a7f8 Migration to yaql 1.0
* Code migrated to yaql 1.0.0
* New MuranoPL object initialization
* Lots of refactoring

See referenced specs for more information

Implements: blueprint migrate-to-yaql-vnext
Implements: blueprint object-construction
Depends-on: I7f314634ab5f08a521e51082d5c84dffca4b0b5c
Closes-Bug: #1454264

Change-Id: I740a4f83c76d8b56a1da585a739d770ef823a524
2015-08-25 16:08:08 +03:00
Kirill Zaitsev
1e62fafe82 Include original ObjectsCopy/Attributes in exception_result
Before if an exception occured during dsl cleanup exception_result would
use empty ObjectsCopy and empty Attributes for result. In case exception
happened during env deletion this would cause env to be deleted by API,
because it ignored isException and treated any result as valid.

Now exception_result also includes original ObjectsCopy/Attributes in case
they're empty in exception_result.
Api not only checks count of 'error' statuses, in session, but also
checks isException attribute of the result, and treats exception results
as Errors, therefore marking deployment/deletion as failed.

Logging of results in API is now aware, that objects can be empty during
app deletion and no longer throws AttributeError because of that.

Change-Id: Idec8191ee25d1cac606741673719bbb8a72709b0
Closes-Bug: #1456724
2015-07-31 16:53:03 +00:00
Ekaterina Chernova
1dd1c24529 Introduces combined class loader
This package class loader combines two types of
class loaders: loading packages from API and from local directory.

If folders to look packages in are specified,
packages would be loaded from their.
Otherwise, standart loader by API will operate as usual.

Implements blueprint change-murano-class-loader

Change-Id: Ifd8f40a755dc580703a44edc2b32cdd17691669d
2015-07-24 16:29:06 +03:00
Nikolay Starodubtsev
de5ed2115a Switch to oslo_log
To follow latest community practices we need to use oslo_log in
murano instead of handcrafted libs.
Unsupported log level 'audit' was changed to 'info'.

Change-Id: I40c0f3790e34bf80d0b63554e86b3cbc019eefca
2015-07-15 11:20:45 +03:00
Sergey Vilgelm
44f4a63642 Declare the CONF variable
In the several places the CONF variable is used from the murano.config.
This patch declares the local CONF variable and uses the oslo_config library.

Change-Id: I1f9a42f7f2cf7fef2cf0d63b185428dfddd9e78e
2015-07-14 00:16:27 +03:00
Stan Lagun
afb730e82d Execute pre/post deployment hooks on GC
Currently pre/post deployment are invoked only for
deployment (actions). Those hooks are used to stop
AgentListener. But they were not executed for GC
phase that could also start AgentListener.  As a result
we could get 2 AgentListeners running and stealing
messages from each other

Change-Id: If96db92e38f067bfed72cfc4a3d5619de2f0fad3
Closes-Bug: #1449500
(cherry picked from commit a79955709d)
2015-05-01 22:43:55 +00:00
Doug Hellmann
21a14acba4 Drop use of 'oslo' namespace package
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.

The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.

Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.

Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages

Change-Id: I975592f3694be42d52685ebf606f6d3012caf1a8
2015-04-28 20:21:22 +00:00
Alexander Tivelkov
f18661b586 Fix for cross-tenant package and class isolation
The database constraints which were present were enforcing the global
uniqueness of package FQNs and the names of classes defined in them.
This behavior was not correct, as the uniqueness should be enforced per
tenant, so the same package may be uploaded into two isolated tenants
without affecting each other.

This behavior lead to a very serious security issue: any tenant could
upload a package, leave it private and thus block all other tenants of
the cloud from uploading the package with the same name or even other
packages which contain at least one class in common with it. This could
be used to intentionally block all the operations of Murano on any
public environments.

This fix modifies the package name constraint to be unique only in
combination with owner_id, i.e. makes packages unique per tenant. Also
it removes the class name uniquness check from database (as there is no
cross-DB way to check it in a proper way) and adds a check method in
db.api module instead.

As the packages may be made public, this introduces a potential
collision: if the user owns some package, and there is a public package
with the same fully-qualified-name (or defining same class(es)) then the
class loader of the engine will have to choise between these packages
and/or classes defined in them.
To resolve this collision this commit adds a logic to fetch all the
patching packages and then pick the best match. Packages owned by the
current tenant are the most preferred, then the engine will pick public
packages, and non-owned non-public packages are the least preferred
(there may be no such packages now, they may appear when we add other
ways of package sharing).

Closes-bug: #1440094
Change-Id: I5c9b49642dfb6e955cf0c98b42f418da3b82060a
2015-04-14 20:58:49 +03:00
Alexander Tivelkov
796a0b2c9d Nova Network support
Adds a support for Nova Network if Neutron is not present in the
current OpenStack deployment.

Supporting the Nova Network requires modifications in three different
parts of generated Heat Stack:
	1) Generated Security Groups and their rules should be of type
	   'AWS::EC2::SecurityGroup', not 'OS::Neutron::SecurityGroup'
	2) Security Group assignments should go to security_groups property
	   of Instance resource, not the network port (as port concept is
	   not present when using NovaNetwork)
	3) FloatingIP should be of type OS::Nova::FloatingIP and should be
	   associated with an Instance by OS::Nova::FloatingIPAssociation
	   resource.

To achieve p1 a SecurityGroupManager class of Core Library is made
abstract and is inherited by two concrete implementations:
NeutronSecurityGroupManager (containing the old MuranoPL code which
generated templates based on OS::Neutron::SecurityGroup) and a new
AwsSecurityGroupManager, which generates AWS-compliant firewall rules
which are consumed by NovaNetwork.

The particular concreate instance of this class is generated by the
default network of environment: Network class has got a new method called
generateSecurityGroupManager which returns an appropriate implementation.

For pp 2-3 a new inheritor of Network class has been added to the Core
Library: an io.murano.resources.NovaNetwork. It generates FloatingIP
association resources if needed and returns a securityGroupName object
as one of the outputs of its joinInstance methods.
The Instance class has been modified to properly handle these types of
outputs.
The instance of the NovaNetwork class is generated at the API side
when a new Environment is created and a is assigned to the
defaultNetworks.environment property of the environment if the neutron
is not defined in keystone.

Also this change moves the auth_utils module from engine to common, as
Keystone Client it contains is now used by the API process as well.

This changed is based on some of the code from the outdated changeset
I6f4b7908bd4bbcd375f64705c7dd06e3954f1ec7

Co-Authored-By: Alexander Tivelkov <ativelkov@mirantis.com>
Co-Authored-By: Stan Lagun <slagun@mirantis.com>

DocImpact

Change-Id: I4c48f33de100a5730ba1d086540d0d99e8fbf9b1
Implements-Blueprint: nova-network-support
2015-04-08 13:09:21 +03:00
Alexander Tivelkov
62c1f10e7b Initial implementation of Plugable Classes
Adds a PluginLoader which loads classes defined as stevedore plugins at
io.murano.extension namespace and registers them as MuranoPL classes in
class loader.

Modifies the ClientManager class to make the _get_client method public,
so other code may use it to add custom clients. This is useful for
plugins which may define their own clients.

Modifies the configuration settings adding 'enabled_plugins' parameter to
control which of the installed plugins are active.

Adds an example plugin which encapsulates Glance interaction logic to:
	* List all available glance images
	* Get Image by ID
	* Get Image by Name
	* Output image info with murano-related metadata

Adds a demo application which demonstrates the usage of plugin. The app
consist of the following components:
	* An 'ImageValidatorMixin' class which inherits generic instance
	  class (io.murano.resources.Instance) and adds a method capable to
	  validate Instance's image for having appropriate murano metadata
	  type. This class may be used as a mixin when added to inheritance
	  hierarchy of concrete instance classes.
	* A concrete class called DemoInstance which inherits from
	  io.murano.resources.LinuxMuranoInstance and ImageValidatorMixin
	  to add the image validation logic to standard Murano-enabled
	  Linux-based instance.
	* An application which deploys a single VM using the DemoInstance
	  class if the tag on user-supplied image matches the user-supplied
	  constant.

The ImageValidatorMixin demonstrates the instantiation of
plugin-provided class and its usage, as well as handling of exception
which may be thrown if the plugin is not installed in the environment.

Change-Id: I978339d87033bbe38dad4c2102612d8f3a1eb3c3
Implements-blueprint: plugable-classes
2015-03-19 14:06:24 +00:00
Dmytro Dovbii
ad6a1da399 Fix pep8 issues in imports
Fixes imports grouping.

Change-Id: I038089cf2d3fc15fefa7d5bc878dad2f16756923
2015-03-13 10:16:08 +02:00
Jenkins
53601f4dcf Merge "Fixes agent call may hanged upon action call" 2015-03-10 08:55:04 +00:00
Stan Lagun
83ba5fa731 Fixes agent call may hanged upon action call
When action called AgentListener automatically starts listening upon
first EP send to the agent. But Environment.deploy() were the only
place where AgentLister was stopped. So when action other than
Environment.deploy() was called there is no one to stop listener.
Thus on each action call new listener on the same RabbitMQ queue
was started causing listeners to steal messages from each other.
Agent.call() that never received response from agent caused
deployment/action hang.

Change-Id: Ia778c816a0e2f57d1f694fd1f128848f61b21a2d
Closes-Bug: #1425963
2015-03-09 08:44:16 +00:00
Stan Lagun
b3e665c63a Makes exception_traceback optional for exception_result
Change-Id: Icd6d655e615fbe92bbf7ba6caaffe5b74c67ffab
Closes-Bug: #1428058
2015-03-05 15:09:42 +00:00
Stan Lagun
73f8368024 Adds API to obtain action result
Also adds File type to core library for common convention type for files

Partially implements: blueprint actions-return-result

Change-Id: I5cbfb9ed6f4ae56e931815841f9c042f25a1d0ca
2015-03-02 14:08:36 +03:00
Ekaterina Chernova
5feedfb8ba Update from global requirements
Change-Id: I955c2945de793ee7f3ce4eb9688e43ec63a6b8e2
2015-02-25 16:09:22 +03:00
Kirill Zaitsev
ceba21573d Include missing log string format specifier
Correctly log non-dsl_exception exceptions
during action execution.

Change-Id: I48d5a3bc033f2830e5bc14a899df49fe7ede014a
2015-02-24 16:24:29 +00:00
Kirill Zaitsev
4e036191ff Use oslo.i18n for translation
Remove gettextutils in favor of oslo.i18n suite for
internationalization purposes. Wrap murano.common.i18n around
oslo.i18n. Mark all logs messages of levels higher than
DEBUG for translation with _/_LI/_LW/_LE/_LC to conform with
oslo.i18n guidelines.

Change-Id: I09a2e2fc802e404f5c59fa4edd2a2124ad24101a
Implements: blueprint organize-translation
2015-02-19 18:51:22 +03:00
ondrej.vojta
a933d6e3b8 Initial implementation of policy enforcement point
Adds ModelPolicyEnforcer that calls congress client (added by commit 2ea56d5b).
Enforcer called only when config property set to true (default false).
Integration test will follow in the next commit (https://review.openstack.org/#/c/147515).

Partially Implements blueprint policy-enforcement-point

Change-Id: Ie53b985ba759c3297e2fe2228bd48fce220ea32f
2015-01-26 16:11:57 +01:00
Ruslan Kamaldinov
734d407332 Replace anyjson with oslo.serialization
Now, when we started to use oslo.serialization it is safe
to replace all the usages of anyjson with jsonutils from
oslo library.
oslo.serialization uses anyjson under the hood, so there
shouldn't be any performance changes.

Change-Id: I8d6fbfbf88e657f5586c7361de849683c064d2e2
2014-12-13 18:10:53 +03:00
Jenkins
af2a623c2b Merge "Use Keystone trusts to get fresh token" 2014-12-10 00:06:51 +00:00
Dmitrii Dovbii
a55acc1266 Remove #noqa from gettextutils imports
Removed #noqa from gettextutils and added them to import_exceptions.
I think it is better to specify option in one place (tox.ini) than
every time take care that you do not forget to specify this tag.

Also removed a few unused imports that were revealed in the process.

Change-Id: Ic4ca9cf374870075a36b88269ff8aea5a8e24a90
2014-11-14 19:43:14 +02:00
Stan Lagun
f40169327b Use Keystone trusts to get fresh token
Instead of using user's auth token (which can expire) for interactions with
other services engine creates Keystone trust that impersonate user and
create new tokens on demand.

Heat stack is created on deployment start using token rather than trust so that
Heat could establish trust of its own (trusts cannot be chained).

New behavior is disabled by default and can be enabled using [engine]/use_trusts = True in murano.conf.
With trusts enabled engine will not work with Heat prior to Juno.
For Heat stacks with deferred actions or long deployment time to work it is also required to turn on trusts in Heat itself.
This can be done via [DEFAULT]/deferred_auth_method=trusts in heat.conf and ensuring that current user
has heat_stack_owner role (or any other that is in [DEFAULT]/trusts_delegated_roles=trusts in heat.conf)

Change-Id: Ic9f3f956ddb6ff2a300a08056ee841cf3c0db870
Implements: blueprint auth-for-long-running-requests
2014-11-14 13:58:17 +03:00
Stan Lagun
56b2d5df27 Fixes silent deletion of environments
Now environment deletion is done as a regular deployment that can fail.
Environments that are deleted, but deletion process has failed remain in database
and shown in dashboard with status 'delete failure'. Environments that are being deleted
has status 'deleting' and do not disappear before they really got deleted on engine side

Also improved status reporting for environments. Now it also reports status of last deployment -
'deploy failure', 'delete failure'

P.S. Functional tests were slightly refactored and fixed to reflect changes
in deletion logic

Change-Id: I05625dd71f7ca9559bb88319b26b122214f15019
Closes-Bug: #1325101
2014-08-19 12:52:52 +04:00
Stan Lagun
705a0f5838 Fixed incorrect information on Python frames in MuranoPL stack traces
Python frames in mixed stack traces were missing file name and pointed to a line
below correct position

Change-Id: I335292f40b3b6ea3dbca80b84f1d8dbed9a6581d
Fixes: bug #1331113
2014-07-12 17:18:33 +00:00
Stan Lagun
c67211da7a MuranoPlException was referenced from incorrect module
Change-Id: I4b10a0e1a0bb3dec99f56d00c621da9898d0725d
2014-06-16 19:59:08 +04:00
Stan Lagun
61f84d03ca Adds ability to throw/catch/rethrow exceptions in MuranoPL
The syntax is
Try:
  - Throw: ns:name   #can be list of names to simulate type hierarchy
     Message: message   #optional
     Cause: $sourceException   #optional
     Extra: { 'someExtra': 'data' }    #optional
Catch:
   - With: ns:name #can be list of names
   - As: exception   #optional
   - Do:
        - Rethrow:
Else:    #optional
    - else block
Finally:    #optional
     - finally block

Improves stack traces to contain information about Python native stack frames and macro blocks

Change-Id: I2e2bcc5e1a0da5f9489d73525f8b3fa99cc0220c
Implements: blueprint muranopl-exception-handling
2014-06-12 20:48:27 +04:00
Stan Lagun
150f06f1b1 Improve method resolution rules for multiple inheritance
With this change MuranoPL can find correct base class method
where old implementation would throw AmbiguousMethodName.

Also removes possibility to have several methods with the same name
but different signature.  This feature didn't worked in most cases,
never used anc could cause unexpected program behavior

Implements: blueprint muranopl-multiple-inheritance-method-resolution

Change-Id: I0a3149b993b1b8a9e9166fce13999e7dd7bf48a5
2014-06-07 09:45:19 +00:00
Serg Melikyan
8c91052b47 Add support for actions in engine
Methods in MuranoPL now can be marked with Usage attribute
to specify whether particular method available for remote
call or not. By default usage is Runtime (not available for
remote call).

Workflow:
  migrateVm:
    Usage: Action
    Arguments:
      - killExisting:
          Contract: $.bool()
          Default: True

Change-Id: If3da3c6bf67aa79d522d82abbf3b5378f72e87ae
Partially-Implements: blueprint application-actions
2014-06-03 14:01:41 +04:00
Stan Lagun
35979b1e02 Preliminary support for HOT packages
Adds support for packages consisting of single HOT template.
Most of HOT features are supported with exception of
environments, attachments and JSON parameter type.

Implements: blueprint hot-packages

Change-Id: I927af0e96f1613e8843ac47844e9c19fa00fdaa6
2014-05-30 12:27:01 +04:00
Ruslan Kamaldinov
86a4c4710d Rename muranoapi to murano
Partially-implements blueprint: rename-murano-api-to-murano

Change-Id: I2d7f52ef4073bce800cedc77f81f5d242c4d5d98
2014-05-27 15:14:44 +04:00