This patch enables the "mcast_flood_reports" and "mcast_flood" (on provnet
ports only) options in the Logical Switch Ports in the OVN driver. Without
these options, the ovn-controller will consume the IGMP queries and won't
send it to the LSP ports, meaning that external IGMP queries will never
arrive to the VMs.
In talks to the core OVN team, it was suggested [0] to enable the
"mcast_flood_reports" option by default in the OVN driver (at least until
fixed in core OVN) as a workaround to this problem. And, to avoid having
to update all ports (which can be many) based on the igmp_snooping_enable
configuration option, we are always setting "mcast_flood_reports" to
"true" in the LSPs. This won't cause any harm (also confirmed by core
OVN developers [0]) since it will be ignored if multicast snoop is
disabled.
[0] https://bugzilla.redhat.com/show_bug.cgi?id=1933990#c3
Closes-Bug: #1918108
Change-Id: I99a60b9af94b8208b5818b035e189822981bb269
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
(cherry picked from commit b04d64b90f)
When a subnet is updated or created, the metadata port is updated too,
to add the fixed IP address of the new subnet. In this case, the port
should update only the IP address of this specific subnet.
Change-Id: I05394e49077a72199bbc80c8cb622ec2b17f2fa7
Closes-Bug: #1890432
(cherry picked from commit 93225e016b)
Setting the ovsdb_probe_interval after Connection.start() is
called means that the probe interval is not changed from
python-ovs's default of 5s until after the initial copy of the
database is retrieved. On busy systems, this can time out and
cause infinite reconnects.
This patch passes the probe_interval argument to the ovs.db.Idl
class so that it can be set as part of creating the jsonrpc
Session.
Some unit tests were removed and replaced with a functional test
which ensures not just that set_probe_interval is called, but that
the value is actually set before the connection is established.
Conflicts:
neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl_ovn.py
Closes-bug: #1905611
Change-Id: I8c940ac8f7632c69607dea7220146ef59d55ed56
(cherry picked from commit 5783e95288)
Prior to this patch the IGMP configuration for ML2/OVN was inconsistent
with the configuration option description and also the ML2/OVS driver
because it was flooding traffic to unregistered VMs [0].
The "igmp_snooping_enable" configuration option says:
"Setting this option to True will also enable Open vSwitch
mcast-snooping-disable-flood-unregistered flag. This option will disable
flooding of unregistered multicast packets to all ports."
But, in ML2/OVN that behavior was inconsistent prior to this patch
because it allowed traffic to flood to unregistered VMs. This patch
fixes it.
Conflicts:
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py
[0]
https://opendev.org/openstack/neutron/src/branch/master/neutron/conf/agent/ovs_conf.py#L36-L47
Change-Id: I5cbe09e26120905b29351d61bbadb30b5dd14938
Closes-Bug: #1904399
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
(cherry picked from commit 9dc8bca740)
Since [1], RowEventHandler._watched_events do not use name mangling.
OvnDbNotifyHandler, inheriting from RowEventHandler, can access to
this variable.
[1]https://review.opendev.org/#/c/752797/
Change-Id: I545c31977799c4ea51e037b631a49306da9c4a3b
Partial-Bug: #1897928
While fixing https://bugs.launchpad.net/neutron/+bug/1876148 we've made
the add_router_interface method idempotent and it no longer raised a
Conflict if the same interface was added twice, this broke the
test_add_interface_in_use from neutron-tempest-pluign.
Change-Id: I76d46a4c3c8af166e022894a5693d5c78555ea50
Closes-Bug: #1895671
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Trivial-ish patch that sets if_exists=True to the pg_del() command when
deleting the port group.
Change-Id: I7c10c8fc2c11329c58fc43b9d611b0aeea2b2ed5
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
After https://review.opendev.org/#/c/745746/ there are new calls to
'with self.lock' which fail in unit tests that only use a Mock object
for the Connection that is passed to an API impl object. Switching to
MagicMock fixes it.
Closes-Bug: #1893965
Change-Id: Icbcb5e004dfa777877d1865a5018262344c7e415
The core OVN team has introduced a new table called Chassis_Private to
avoid nb_cfg flooding when checking for the Chassis' status. The OVN
driver does rely on that mechanism for the agent liveness mechanism.
This patch makes use of this new table but it's also backward
compatible.
For more information, check the core OVN changes at:
https://patchwork.ozlabs.org/patch/1254394.
Closes-Bug: #1892477
Change-Id: Iea4263b852d1e3f81eb2557918ea3cbb7adb8016
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This is a subset of the changes for implementing the floating IP
port forwarding feature in neutron, using OVN as the backend.
This changeset updates ovn_db_sync utility and its tests to
ensure that floating ip port forwarding can be repaired.
Depends-On: https://review.opendev.org/#/c/741303/
Change-Id: I7a158173252e73e081914f242133634c41de7999
Partially-implements: ovn/port_forwarding
Partial-Bug: #1877447
OVN firewall driver can't silently normalize CIRDs given in
the security group rule's "remote_ip_prefix".
Because of that if user created rule with not normalized CIDR, it
wasn't applied by the OVN driver.
Now OVN driver will normalize such rules before applying them.
The OVN driver will now also check if SG rules with same normalized
and same direction, port range, protocol and ethertype already exists in
the SG. If so, it will not add or remove rule in the OVN.
Rule will be added or removed only if there is no other same rules in
the SG.
Change-Id: I0d9295545384844e81b0ffe3aa7483324f9a9ae5
Related-Bug: #1869129
This is a subset of the changes for implementing the floating IP
port forwarding feature in neutron, using OVN as the backend.
This changeset covers the additions to northbound api needed for
handling load balancer entries created on behalf of port forwarding.
Depends-On: https://review.opendev.org/#/c/729354/
Change-Id: I7d0aa51468dbd2c298395c3dccd8f222e87032d8
Partially-implements: ovn/port_forwarding
Partial-Bug: #1877447
By raising an exception when an option is invalid we broke the Ironic +
OVN + Neutron DHCP agent combination that enabled deploying baremetal
machines.
This patch is changing the approach to just log the invalid options
instead of failing the request so that the OVN driver can still be used
with Ironic.
Change-Id: I5e98297acefb62f9a9c1200ccfaac0672eeeed2c
Closes-Bug: #1888649
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
The general usage of verify() is if you are doing something like:
ports = row.ports
ports.append('newport')
row.ports = ports
where you read a value from the db, modify it, then overwrite it.
The setkey/addvalue/delvalue mutate functions make it largely
unnecessary to use verify(), so this patch removes those.
Change-Id: Idd42491dcaa10ec36c963b477438eaa2336ef3a0
OVN creates localport [1] for each network that has metadata
and allocate IP address from subnet within this network that has
DHCP enabled. The traffic from this port will never go outside
the chassis.
While using multiple segments with subnet linked to each segment
OVN needs to create an allocation of IP address for each of those
subnets [2] in order to generate data for OVN NBDB IPv4 DHCP Options.
The change [3] started to validate that condition, while multiple
IP addresses from different segments are tried to be allocated on
one port. We can skip this for OVN Metadata port, because there
is no reason to prevent those kind of allocation for OVN.
[1] http://www.openvswitch.org/support/dist-docs/ovn-architecture.7.html
[2] 5f42488a9a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py (L2279)
[3] https://review.opendev.org/#/c/709444/
Change-Id: Ib51cde89ed873f48db4daebc27a0980da9cc0f19
Closes-Bug: 1871608
While port has QoS policy configured the policy wasn't deleted
because of logic issue.
Change-Id: I3d7e70a4a110c68a89d6c575abf121cd9b97e439
Closes-Bug: #1886962
OVN distributed services like Metadata and DHCP uses now
DEVICE_OWNER_DHCP device_owner which isn't distributed by its nature.
To fully use benefits of OVN Distributed ports (localports) [1]
and to not overlap with Neutron logic created for not-distributed
ports we should use new device_owner.
In this change we need also to bump minimum required
neutron-lib to 2.4.0.
[1] https://www.ovn.org/support/dist-docs/ovn-nb.5.txt
Change-Id: I0a69f1bddaa7030c7287216e62ec1ac6dd381475
OVN's API called get_port_groups is poorly named and has misleading docstring.
It returns only the OVN port groups that map to the security group in Neutron.
Therefore, it should be called get_sg_port_groups.
Closes-Bug: #1883716
Related-Bug: #1881316
Change-Id: Iae3f413dd1c4b0813b05d9bfd593c9e709540370
Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
This patch is adding support for the router_availability_zone extension
for Neutron.
The OVN driver will now read from the router's availability_zone_hints
field and schedule the router ports onto OVN chassis belonging to those
AZs.
Since the OVN driver does not rely on the L3 agent, this patch does not
re-use the configuration option for the agent to configure the
availability zone that a Chassis belongs to (even because there's no
configuration file in nodes such as networker nodes). Instead, this
patch reuses the "ovn-cms-options" field from the local OVSDB to
configure the Chassis. The follow syntax has been used:
$ ovs-vsctl set Open_VSwitch .
external-ids:ovn-cms-options="enable-chassis-as-gw,availability-zones=az0:az1"
In the example above, the Chassis has been configured to belong to two
AZs: "az0" and "az1".
This patch also implements listing the availability zones:
$ openstack availability zone list
As well as validating the router's availability zone hints:
$ openstack router create --availability-zone-hint az0
--availability-zone-hint az1 test_router
The above command would fail if there's no "az0" and "az1" configured in
any OVN chassis.
Documentation for this feature is being written and will be submitted
in a separated patch.
Partial-Bug: #1881095
Change-Id: I4567f3d541d382b6432c1ab3d35276d81ce71d82
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
As stated in the bug description, there are many writes of the
agent liveness external_ids into the Chassis table. There was a
protection to avoid bumping nb_cfg too frequently.
The same protection is reused to avoid writing into the Chassis
external_ids.
This patch reduces the number of transactions to the SB database
and, therefore, the recomputations that it causes to ovn-controller
in all nodes.
Change-Id: I5db90fde8e7394772ec23c6384c711096c246621
Closes-Bug: #1883554
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
Prior to this patch, if the [ovn]/enable_distributed_floating_ip
configuration option changed in an existing environment the OVN
driver wouldn't adapt to it requiring administrators to clear up the
"external_mac" column from the NAT table manually for the existing
floating ips.
With this patch, OVN will automatically correct existing NAT entries for
floating ips whenever this option changes.
To make the code simpler when handling the port up/down event this patch
always set the logical_port and the neutron:fip_external_mac key in the
external_ids column of the NAT entry when creating the floating ip.
Note that we are not using the maintenance task for this either, we are
re-using the event that set/unset the "external_mac" column for this
because, whenenver the service is restarted (after the configuration is
changed, we need to restart for it to take effect) the IDL will re-trigger
those events.
Closes-Bug: #1883559
Change-Id: I6a85fdde2558d781bf2853c5d11c5c964bbab81f
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This patch removes unused UpdateACLsCommand which was previously used by the
code using Address Sets but that has been already removed. OpenStack API
doesn't implement update operation for security group rules so this
command is not needed anymore.
Change-Id: Iea4a82a21d32a064ff5dd530e104a2de2efb46b5
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Port_Groups table has been introduced in OVN 2.10 and we've moved in
master to newer version since. This patch removes all references to code
branching between port_groups and address_sets, and also removes
unneeded Address_Set commands and references.
Change-Id: I592d31db9be76d9be202d79d942e15b1668e3c0e
If new segment is created/old deleted we should update its
localnet port in related Logical_Switch.
Added also missing code to sync tool in order to delete provnet
ports in case of leftovers.
Change-Id: I6b864ba1c168643640a64bd7c25e1d0fc0ea348a
Related-Bug: 1865889
Prior to this patch OVN did not validate any extra DHCP option passed
to the port leading to confusion because the user of the API could just
input any value and OVN would accept it (returning 200) but ignoring the
option internally.
This patch now adds such validations on port creation and update.
This patch also sync with the latest supported DHCP options from OVN and
create a map between the different names and option codes to their OVN
counterpart.
Closes-bug: #1874282
Change-Id: I99799e54e941cdd8da2614feecad1ef6299703fc
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
The periodic job running ovsdbapp master started failing after the
ovsdbapp custom indexing patch merged. There were just some things
that needed to be mocked out.
Closes-Bug: #1879717
Change-Id: I37d4c655662dd9d55b6096f2dc4fc861894706d2
OVNL3RouterPlugin inherits from L3_NAT_dbonly_mixin, which inherits
from neutron.extensions.l3.RouterPluginBase
As maintenance task expects OVNL3RouterPlugin to behave as
RouterPluginBase, the add_router_interface should have the signature:
add_router_interface(self, context, router_id, interface_info)
Note: With this change, the default behavior of OVNL3RouterPlugin's
_add_neutron_router_interface becomes idem-potent: multiple calls to add
the same interface will not fail. Because of that, the unit test
test_router_add_interface_dup_port no longer makes sense and is being
removed.
Closes-Bug: #1876148
Change-Id: I8010113b4d8c66ecbccf3126f322a8836d92e7ba
Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
The patch adds a short living connection in pre-fork routine that
creates neutron_pg_drop Port Group. Later after workers are spawned,
each worker also creates a short living connection and waits for an
event that the Port Group was created.
The short living IDLs limit its tables only for relevant tables so it
doesn't fetch the whole OVS DB to the local copy.
Closes-bug: #1866068
Change-Id: I1f5af36b8c3d5650f890edfed3c33dc206869824
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
The delete_port() method from OVNClient has a potential problem of
leaving stale ports when RowNotFound is raised from the process to
delete the port from the OVN database. Since the exception is not
granular enough, the RowNotFound could be raised from other objects that
are part of the same transaction (such as ACLs, DNS entries, etc...)
resulting in the revision for the port being deleted even tho the port
is still in the database.
Instead of giving a pass on the RowNotFound exception, this patch is
logging the error and re-raising it without deleting the revision.
Change-Id: I25b93b7c080403fc38365b638e4e03298b447d0f
Partial-Bug: #1874733
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Prior to this patch, the OVN driver wasn't account for the VNIC types
VNIC_DIRECT_PHYSICAL and VNIC_MACVTAP. These types should work the same
way as the VNIC_DIRECT type in the OVN driver perspective.
Closes-Bug: #1874065
Change-Id: Idb596b5a80a3155bc9cdee1e082506701e730f00
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
The QoS OVN client extension is moved to the ML2 driver. This
extension is called from the OVN driver in the events of:
- create port
- update port
- delete port
- update network
The QoS OVN client extension now can accept several rules per policy
as documented in the SUPPORTED_RULES. The QoS OVN client extension
can write one OVN QoS rule per flow direction and each OVN QoS rule
register can hold both a bandwidth limit rule and a DSCP marking rule.
The "update_policy" method is called from the OVN QoS driver, when
a QoS policy or its rules are updated.
The QoS OVN client extension updates the QoS OVN registers
exclusively, based on the related events.
Closes-Bug: #1863852
Change-Id: I4833ed0c9a2741bdd007d4ebb3e8c1cb4c30d4c7
Only reschedule gateways/update segments when things have changed
that would require those actions.
Co-Authored-By: Terry Wilson <twilson@redhat.com>
Change-Id: I62f53dbd862c0f38af4a1434d453e97c18777eb4
Closes-bug: #1861510
Closes-bug: #1861509
We can now revert this patch, because main cause has been already
fixed in Core OVN [1]. With this fix the ARP responder flows are not
installed on LS pipeline, when LSP has port security disabled, and
an 'unknown' address is set in addresses column.
This makes MAC spoofing possible.
[1] https://patchwork.ozlabs.org/patch/1258152/
This reverts commit 03b87ad963.
Change-Id: Ie4c87d325b671348e133d62818d99af147d50ca2
Closes-Bug: #1864027
The "old" parameter passed to the handle_ha_chassis_group_changes()
method is a delta object and sometimes it does not contain the
"external_ids" column (because it hasn't changed).
The absence of that column was misleading that method into believe that
the "old" object was no longer a gateway chassis and that triggered some
changes in the HA group. Changing the HA group resulted in the SRIOV
(external in OVN) ports to start flapping between the gateway chassis.
This patch is adding a check to verify that the "external_ids" column
has changed before acting on it, otherwise just ignore the update and
return.
Closes-Bug: #1869389
Change-Id: I3f7de633e5546dc78c3546b9c34ea81d0a0524d3
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
The check _is_virtual_port_supported() prevented us from
clearing the addresses field while port was OVN LB VIP port.
The virtual port should be set only when port is Octavia Amphorae
VIP port.
Change-Id: Id6dd29650951855d13498a7206f6d1dde7db7864
Closes-Bug: 1863893
oslo-utils "isotime" is deprecated [1]. "datetime.datetime.isoformat"
should be used instead.
[1]382370781b/oslo_utils/timeutils.py (L45-L49)
Change-Id: Iaaab299298c6528ea56e4b212674f492dfe517b7
The OVN maintenance code was not always calling into
the OVNClient class methods with the correct number of
arguments, leading to exceptions.
After a deeper review, there were a number of places
where this was happening, so changed most methods to
take a 'context' argument since it's usually available
in the caller.
Change-Id: I1bcb0ca68747e4c32523e41307dc132291c55f6d
Closes-bug: #1861502
This patch introduces a new mechanism to allow rerunning maintenance
tasks upon an OVN database schema change to avoid a service restart.
As an example, the "migrate_to_port_groups" maintenance task will run
again when the database schema is updated. In case of a migration from
an OVN version without port groups support to a version that supports
it, the OVN driver will migrate the code automatically without the need
of a service restart.
Closes-Bug: #1864641
Change-Id: I520a3de105b4c6924908e099a3b8d60c3280f499
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This patch is adding support for a new port type called "external" in
core OVN.
Prior to this work, when a VM had a SR-IOV port attached to it, OVN itself
wasn't able to reply to things such as DHCP requests packets since the
OVS port was skipped. Core OVN then introduced the concept of "external"
ports which are ports deployed on a different node than the one that the
VM is running and is able to reply to such requests on behalf of the VM.
With this patch, when a port with the VNIC type "direct" and no
"switchdev" capability is created, ovn driver will then create a
logical port with the type "external" for it and add it to a default
HA Chassis Group. The port will then get bound to the "master" (higher
priority) chassis of that group.
Please note that, as a first step, this patch is creating only one HA
Chassis Group which *all* external ports will belong to. That means that
all external ports will be *scheduled onto the same node* (but it's
HA nevertheless). In the future we should enhance this behavior.
Change-Id: Ic6c4bb6c584682169f3ebd73105a847b05dddc76
Closes-Bug: #1841154
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This patch is ading IGMP snooping support in the OVN driver. Multicast
support has been introduced in core OVN upstream.
Also, the patch always sets the "mcast_flood_unregistered" config in
the OVN Logical_Switch to the same value as the "mcast_snoop" config.
This is so that OVN matches the OVS behavior which is to enable IGMP
flooding by default [0] (in OVN, by default it's false).
[0] http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.txt (grep
for "mcast-snooping-disable")
Change-Id: I32f61ba3dd06d7eacf76a74c5c44e1286f90e584
Co-Authored-By: Daniel Alvarez <dalvarez@redhat.com>
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This patch adds a minimum interval between each agent health checks.
The way OVN checks for the agents liveness is by increasing a value in
the NB DB and waiting for it to be propagated to the SB DB but, this can
be costy if done many times too quickly. Therefore, a minimum interval
between each check is being added.
Closes-Bug: #1861092
Change-Id: If1f2d97e3a3a17f6744d546b3e8903bde55e83b9
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Right now neutron-server bumps the nb_cfg parameter in NB_Global
table which needs to be propagated by northd to SB_Global,
processed by agents, and write it back into SB_Global.
This requires processing by neutron-server but unfortunatelly
the server checks straight away and many times the value read
is behind the expected value.
All this results in frequent false positives showing dead agents
when they are not.
This patch is relaxing the checks by allowing a difference of 1
between the read and expected values.
Change-Id: Id91481b690ad569c5dcfa5bd404f497f591d729d
Closes-Bug: 1860436
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
This context was required but not passed. This change adds it.
Change-Id: I5e5e14d5a621fc4a790810d15e1162f90898282e
Related-Blueprint: neutron-ovn-merge
Previous to this patch, the 'unknown' address was added to the
OVN Logical_Switch_Port row. However, the '<ip> <mac1> <macN>'
was still there so the port security was not really disabled.
The correct behavior is to set the 'addresses' field to 'unknown'
and remove everything else.
Change-Id: I0b84f41865e3fdea49cf169df5431249c35f5ff8
Closes-Bug: #1728886
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
Co-Authored-By: Richard Theis <rtheis@us.ibm.com>
Co-Authored-By: John Kasperski <jckasper@us.ibm.com>
Co-Authored-By: Numan Siddique <nusiddiq@redhat.com>
Co-Authored-By: Terry Wilson <twilson@redhat.com>
Co-Authored-By: Lucas Alvares Gomes <lucasagomes@gmail.com>
Related-Blueprint: neutron-ovn-merge
Change-Id: I62e12c1319f4f805c790b33963a76c5d7ea79d07
This is a tweak to changes done to fix bug 1834637. Specifically,
this change addresses scaling. The previous gerrit change had
modifications to all OVN sub-ports performed as a single
transaction. That did not account for race-condition on neutron
DB queries, which leads to timeouts under heavy loads.
Another cleanup done by this change is to fold an additional
update on neutron db into ovn trunk driver. That saves
update from doing another database transaction, thus making
it faster. The no longer needed function in mech_driver was
called _update_subport_host_if_needed
By breaking the iteration into multiple transactions, the
change in time is marginal:
Service-level agreement
NeutronTrunks :: neutron.create_trunk
from 34.2 sec to 35.6 for 50%ile
from 35.6 sec to 36.1 for 95%ile
This patch is a cherry pick from networking-ovn stable/train [1], that
is now part of networking-ovn migration to neutron repo.
[1]: https://review.opendev.org/#/c/698863/
Change-Id: I118f93cdd34a1d25327a0dc61367720ac6559001
Closes-Bug: #1834637
Co-authored-by: Maciej Józefczyk <mjozefcz@redhat.com>
(cherry picked from commit c418dd720b9be9047d779f32407c474e90cb0002)