966 Commits

Author SHA1 Message Date
Zuul
fa2500c15c Merge "Use maven for simple_json jar" 2019-11-13 01:08:01 +00:00
Tim Burke
ccef5d4dfe Update test_deploy_storlet expectations to be bytes
...since we didn't provide any encoding. Also, open files in binary mode
when hashing to compare against ETags.

Change-Id: Ie8ac1d1ff0216d945b1b1602c944cb62234a84a8
2019-11-12 14:17:49 -08:00
Zuul
ade5040b82 Merge "Ensure task_id is a native string on py3" 2019-11-11 03:07:51 +00:00
Zuul
ed131d9ba4 Merge "PY3: Fix datagram message to be bytes" 2019-11-11 03:04:50 +00:00
Zuul
69cff66d8e Merge "Port gathering logs routine from .functests to post ansible yaml" 2019-11-11 02:55:20 +00:00
Tim Burke
2e5823f007 Clean up tmp file in more cases
Previously, if the test failed, we would not remove the file.

Change-Id: Icaa3f2894feab48279073ddb19e8586f0528763b
2019-11-09 02:02:41 +00:00
Tim Burke
e4363f4d1f Use maven for simple_json jar
Google is not accessible from China.

Change-Id: I8eaee3954cd480acca11c76f3b5dc8080b7fa8d9
2019-11-09 01:59:08 +00:00
Tim Burke
01e3dfa92f py3: Fix up some func tests
Lots of str-needing-to-be-bytes fixes, mostly.

Seems to be an issue with tests around executing? Skipped the two that
would just hang, but test_deploy_storlet might be a good place to start
poking around.

Change-Id: I99a6beda9c3a8883d9f13a953cba13aa21d847bb
2019-11-08 23:04:31 +00:00
Tim Burke
9f2e9b20b8 Make jupyter test notebook py2/3 compatible
Change-Id: I2649728d5a240e0f476f71742f4e950b4c68e643
2019-11-08 14:39:21 +00:00
Tim Burke
ba5aa7cdb8 Ensure task_id is a native string on py3
Change-Id: I4f4ae8f9d1248feaca5b9181fc72e5b44e1eed4e
2019-11-08 06:40:14 +00:00
Kota Tsuyuzaki
236aaf2894 PY3: Fix datagram message to be bytes
And the reply should be str after it was received.

Change-Id: I69ebd60bda4e711b560b8cc7b3487988f69acafa
2019-11-08 06:33:59 +00:00
Zuul
a4087219c7 Merge "PY3: skip jupyter notebook test" 2019-11-08 06:10:39 +00:00
Kota Tsuyuzaki
28677c0113 Port gathering logs routine from .functests to post ansible yaml
Change-Id: I5cc87cac7eb2a36b8c86a4807592d3ee64d94780
2019-11-08 14:57:57 +09:00
Zuul
cda48e29ca Merge "Decrease fuctional test timeout" 2019-11-08 05:31:01 +00:00
Zuul
97d9a7fea2 Merge "Fix CI to collect the all logs" 2019-11-08 04:13:42 +00:00
Kota Tsuyuzaki
839d1750a7 Decrease fuctional test timeout
Due to the change https://review.opendev.org/#/c/693309/
the functional test takes around 20 minutes-ish so that
we can decrease the timeout as 30 minutes.

Change-Id: Ic82decc34e5fa5a4049a08b95a8e8f606d9cd973
2019-11-08 13:10:11 +09:00
Kota Tsuyuzaki
8ec7ae5edc PY3: skip jupyter notebook test
because the notebook is strictly tightened to python2.
Re-writing the python2 notebook in python3 will cause
an error or a failure in python2 functional test, so
add skipTest check for now.

In the future, we could switch the available version
or have a couple of tests for both versions.

Change-Id: I9ec5cc107459266de509d3c444c5294f965dd749
2019-11-08 12:51:25 +09:00
Zuul
53ff2b7a84 Merge "Migrate detailed implementation from __init__.py" 2019-11-08 03:24:13 +00:00
Zuul
da381abd4d Merge "Use smaller file for functional tests" 2019-11-08 03:23:10 +00:00
Kota Tsuyuzaki
b0aa01f116 Fix CI to collect the all logs
Currently, the CI may loose the swift proxy error logs
if previous task collecting the func test tox logs failed.
That makes us hard to dig out what happens actually.

This patch attempts to ignore the error and reach out whole
worthful tasks in the post job.

Change-Id: I8a0e22f57255ee7b011d660e19e4ffa664b94d23
2019-11-08 11:56:18 +09:00
Zuul
9696fe43a2 Merge "Be explicit about which python3 to use for py36 and py37 tox envs" 2019-11-08 02:38:05 +00:00
Takashi Kajinami
dfbdd27bd8 Use smaller file for functional tests
1GB and 512MB are too large to test chunked read, and 10MB should
be enough.

Change-Id: Id91acedbc144913769d0b5ced6a6acfc3268d826
2019-11-08 11:11:10 +09:00
Takashi Kajinami
8e31cdc7ba Migrate detailed implementation from __init__.py
... as we don't expect long codes are included in __init__.py .
This patch migrates these implementations to new independent
files, with keeping alias so that we can import things with
the same path.

Change-Id: Iff0b60cd53281c999c930a4e789aaaab360b1c7a
2019-11-08 11:07:39 +09:00
Kota Tsuyuzaki
c72226b654 PY3: decode subprocess output if it's bytes
Otherwise, the logger will cause errors for the output.

Change-Id: I4379dba7629cfc0d928c1adda587ca400a38d227
2019-11-07 23:49:06 +00:00
Kota Tsuyuzaki
89fd96b9ff PY3: ensure bytes format in sbus conversation
Originally, that was just str, that can be compatible with
char in python c_types. However, in python3, str is not a
byte format but bytes. This patch is intended to encupsulate
any data format (path name, metadata) into SBus module that
tranforms them from str to bytes at the send time. (receive
will work as well from bytes to str) Thus, any other storlet
modules are able to use them as str as well as ever.

Change-Id: I542db555798e7ef67bf0db9d98d1c14578118f86
2019-11-07 23:49:06 +00:00
Tim Burke
3369c8b613 Be explicit about which python3 to use for py36 and py37 tox envs
I saw some warnings locally like

    /usr/local/lib/python3.6/dist-packages/tox/config/__init__.py:581:
    UserWarning: conflicting basepython version (set 3.6, should be 3.7)
    for env 'py37';resolve conflict or set ignore_basepython_conflict

Seems like that may happen in the gate, too?

Change-Id: I31f12bedf77163c8756fecb5000885d01ddbf0e6
2019-11-07 23:46:54 +00:00
Kota Tsuyuzaki
84421c93d0 PY3: Fix file open to read/write the data as bytes
Because swiftclient.put_object assumes it as bytes object.
Only ZipFile is not suppoting rb mode so leave it as it is.
It should be okay because Storlets use it just to pick up
the filnames that should be string instead of bytes.

Change-Id: I834b29886a18269de2745ffe584291491e80f9eb
2019-11-07 16:47:27 +00:00
Zuul
77573c5906 Merge "Switch to Ussuri jobs" 2019-11-07 08:22:53 +00:00
Zuul
bf0dec8020 Merge "Enable unit tests related to internal_client" 2019-11-07 07:50:21 +00:00
Takashi Kajinami
12bc89be86 Enable unit tests related to internal_client
These tests are skipped because we use swift codes which weren't
compatible with Py2, but now we are ready to use them in Py3.

Change-Id: I717686c7d63bb0b21d25f534ba439e611ea2f5bc
2019-11-07 14:52:41 +09:00
Kota Tsuyuzaki
9eb84a2282 PY3: Fix urllib path to be compatible with py3
Change-Id: If6a41b27dcda4c556d9b0ca7da58b847dc606f5c
2019-11-01 17:56:29 +09:00
pengyuesheng
926b4bc64e Switch to Ussuri jobs
Change-Id: Ifa7ac6f86219001c692d713a006c91bf593f6280
2019-10-25 15:11:33 +08:00
Zuul
e5f992afac Merge "Update the constraints url" 2019-10-25 02:43:18 +00:00
pengyuesheng
a05c782660 Sync Sphinx requirement
Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.

Change-Id: Idbd27e14fbb70278fb1a63e3d0b2a6456b615e41
2019-10-10 15:07:34 +08:00
Zuul
3f32fcef7a Merge "Update master for stable/train" 2019-10-08 05:14:14 +00:00
Kota Tsuyuzaki
a543cd3396 Add release note for 4.0.0
Change-Id: Ic778cdc20ca30c8832de91956e4d0d60aebdb163
2019-10-03 13:29:45 +09:00
04ed79a7e3 Update master for stable/train
Add file to the reno documentation build to show release notes for
stable/train.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.

Note that release note for train doesn't seem to be ready.
(no 4_0_0_0_hash file exists)

Change-Id: Ia1f9e21af0eab90e1685e9e482fd8affcc415989
Sem-Ver: feature
2019-10-02 14:04:17 +09:00
pengyuesheng
9097e05704 Update the constraints url
For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Change-Id: Ida252ab192dee528ddcaf401ece57f66c0e734f8
2019-09-29 10:44:15 +08:00
Takashi Kajinami
18b05fe4bd Merge file descriptors and their metadata into one class
This patch bundiles a file descriptor and its metadata
into one class named SBusFileDescriptor, so that we can
keep the relationship between them easily.

Change-Id: Ib042f2a7bd4bd524e9b4b8d155ec192edaaca7f0
2019-09-25 19:46:02 +09:00
Takashi Kajinami
c11d067027 Drop SBUS_CMD_DESCRIPTOR
SBUS_CMD_DESCRIPTOR was implemented in java SDaemon, but
we have never used it.
This patch drop that sbus command, as it is never used and
tested.

Change-Id: Iac92e598ddaa027b9882fbcb8be8ff9738226801
2019-09-25 19:46:02 +09:00
Takashi Kajinami
fa433a44c1 Refactor java written SDaemon
This patch refactors java written SDaemon, by migrating
task specific logics from SDaemon to each Task classes.

*bonus* eliminate trailing white space that can not be
found by pep8 (because it's Java code!)

Change-Id: Ife21fe368a17428dab27cf245ada4f3c0f9f1278
2019-09-25 19:46:02 +09:00
Kota Tsuyuzaki
92e0e58d7f Update functional test timeout
because it looks zuul takes 25 minutes for preparation
so we have to increase timeout to avoid unnecessary failures

Change-Id: I5fb291eadc4e49e19a225e788de564d38a43fbfe
2019-09-25 17:51:04 +09:00
Zuul
1ec8a7bc9c Merge "Create an independent directory to place jar libraries" 2019-09-25 04:57:05 +00:00
Zuul
e2965d6082 Merge "Normalize method names for RunTimePaths class" 2019-09-25 03:04:19 +00:00
Zuul
59b0175275 Merge "Use json for response sent through service out fd" 2019-09-25 02:35:56 +00:00
Takashi Kajinami
c27c1f894b Fix inappropriate arg passed to nostest command
Change-Id: I2d98115c1fb43f9b6faeb529d42092f676629de3
2019-09-05 23:32:44 +09:00
Takashi Kajinami
4450bf1e23 Normalize method names for RunTimePaths class
This patch normalizes method name for RunTimePaths, according
to the following policy, so that it is more easy to understand
the characteristics of each method functions

 - Use @propery for the functions without any arguments
 - Use get_foo for the functions which take arguments
 - Use foo_dir for directory paths

This patch also fixes wrong path handling in storlet app log.

Change-Id: I9275a84a2881b29a91eb55efaa6bc5607f1e723c
2019-08-25 22:41:10 +09:00
Takashi Kajinami
8188824218 Fix build warning in c codes
There are some build warning observed when building c codes.
It is not critical as they are still warning, but it would be
better to avoid these warning if possible.

Trivial-fix
Change-Id: I5f89fc0defce43984471576025636e172630e733
2019-08-25 00:53:29 +09:00
Zuul
ecbeae721b Merge "Remove unused function" 2019-08-18 09:59:42 +00:00
Takashi Kajinami
ec9b0406ad Remove unused function
This patch drops _open_pipe function, which is not used in anywhere.

TrivialFix

Change-Id: I6c54e5783898ea997e53ba023df7a8d675920965
2019-08-18 17:31:37 +09:00