openstack/sahara-extra
Code Issues Proposed changes

Retire Sahara: remove repo content

Browse Source 
Sahara project is retiring
- https://review.opendev.org/c/openstack/governance/+/919374

this commit remove the content of this project repo

Depends-On: https://review.opendev.org/c/openstack/project-config/+/919376
Change-Id: I4ff53b361a24b624048ba013861c7fcf51997010
This commit is contained in:
Ghanshyam Mann 2024-05-10 17:27:40 -07:00
parent 1d0bdedd51
commit bbe243e2bf
133 changed files with 8 additions and 16502 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
.gitignore vendored
View File

@ -1,28 +0,0 @@
*.py[co]
*.egg
*.egg-info
dist
build
eggs
parts
var
sdist
develop-eggs
.installed.cfg
pip-log.txt
.tox
*.mo
.mr.developer.cfg
.DS_Store
Thumbs.db
.venv
.idea
out
target
*.iml
*.ipr
*.iws
*.db
.coverage
ChangeLog
AUTHORS

4
.mailmap
View File

@ -1,4 +0,0 @@
# Format is:
# <preferred e-mail> <other e-mail 1>
# <preferred e-mail> <other e-mail 2>
Ivan Berezovskiy <iberezovskiy@mirantis.com>

40
.zuul.yaml
View File

@ -1,40 +0,0 @@
- project:
templates:
- check-requirements
check:
jobs:
- sahara-extra-build-artifacts:
voting: false
- openstack-tox-pep8
gate:
jobs:
- sahara-extra-build-artifacts:
voting: false
- openstack-tox-pep8
post:
jobs:
- sahara-extra-publish-artifacts
- publish-openstack-python-branch-tarball
- job:
name: sahara-extra-build-artifacts
description: |
Build the artifacts used by Sahara clusters and jobs.
parent: unittests
timeout: 3600
required-projects:
- openstack/sahara-extra
run: playbooks/build-artifacts/run.yaml
- job:
name: sahara-extra-publish-artifacts
description: |
Build the artifacts used by Sahara clusters and jobs
and publish them on tarballs.openstack.org
parent: publish-openstack-artifacts
timeout: 3600
final: true
required-projects:
- openstack/sahara-extra
run: playbooks/build-artifacts/run.yaml
post-run: playbooks/build-artifacts/post.yaml

19
CONTRIBUTING.rst
View File

@ -1,19 +0,0 @@
The source repository for this project can be found at:
https://opendev.org/openstack/sahara-extra
Pull requests submitted through GitHub are not monitored.
To start contributing to OpenStack, follow the steps in the contribution guide
to set up and use Gerrit:
https://docs.openstack.org/contributors/code-and-documentation/quick-start.html
Bugs should be filed on Storyboard:
https://storyboard.openstack.org/#!/project/openstack/sahara-extra
For more specific information about contributing to this repository, see the
sahara contributor guide:
https://docs.openstack.org/sahara/latest/contributor/contributing.html

12
HACKING.rst
View File

@ -1,12 +0,0 @@
Sahara Style Commandments
=========================
- Step 1: Read the OpenStack Style Commandments
https://docs.openstack.org/hacking/latest/
- Step 2: Read on
Sahara Specific Commandments
----------------------------
None so far

175
LICENSE
View File

@ -1,175 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

2
MANIFEST.in
View File

@ -1,2 +0,0 @@
include README.md
graft elements

27
README.rst
View File

@ -1,21 +1,10 @@
========================
Team and repository tags
========================
This project is no longer maintained.
.. image:: https://governance.openstack.org/tc/badges/sahara-extra.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
.. Change things from this point on
OpenStack Data Processing ("Sahara") extra repo
===============================================
Sahara-extra is place for Sahara components not included into the main `Sahara repository <https://opendev.org/openstack/sahara>`_
Here is the list of components:
* Sources for Swift filesystem implementation for Hadoop: https://opendev.org/openstack/sahara-extra/src/branch/master/hadoop-swiftfs/README.rst
* Sources for main function wrapper that adapt for oozie: https://opendev.org/openstack/sahara-extra/src/branch/master/edp-adapt-for-oozie/README.rst
* Sources for main function wrapper that adapt for spark: https://opendev.org/openstack/sahara-extra/src/branch/master/edp-adapt-for-spark/README.rst
* `Diskimage-builder <https://opendev.org/openstack/diskimage-builder>`_ elements moved to the new repo: https://opendev.org/openstack/sahara-image-elements
* Tools for building artifacts: https://opendev.org/openstack/sahara-extra/src/branch/master/tools
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
OFTC.

BIN
common-artifacts/US_export_policy.jar
View File

Binary file not shown.

BIN
common-artifacts/ext-2.2.zip
View File

Binary file not shown.

BIN
common-artifacts/hadoop-aws-2.7.1.jar
View File

Binary file not shown.

BIN
common-artifacts/hadoop-aws-2.7.3.jar
View File

Binary file not shown.

BIN
common-artifacts/hadoop-aws-2.7.5.jar
View File

Binary file not shown.

BIN
common-artifacts/hadoop-native-libs-2.7.1.tar.gz
View File

Binary file not shown.

BIN
common-artifacts/hadoop-native-libs-2.7.5.tar.gz
View File

Binary file not shown.

BIN
common-artifacts/hadoop-native-libs-2.8.2.tar.gz
View File

Binary file not shown.

BIN
common-artifacts/hadoop-native-libs-3.0.1.tar.gz
View File

Binary file not shown.

BIN
common-artifacts/local_policy.jar
View File

Binary file not shown.

24
edp-adapt-for-oozie/README.rst
View File

@ -1,24 +0,0 @@
=======================
Sources for main function wrapper that adapt for oozie
=======================
In order to pass configurations to MapReduce Application through oozie,
it is necessary to add the following code.
(https://opendev.org/openstack/sahara-tests/src/branch/master/sahara_tests/scenario/defaults/edp-examples/edp-java/README.rst)
// This will add properties from the <configuration> tag specified
// in the Oozie workflow. For java actions, Oozie writes the
// configuration values to a file pointed to by ooze.action.conf.xml
conf.addResource(new Path("file:///",
System.getProperty("oozie.action.conf.xml")));
This wrapper adds a above configuration file to a default resources and
invoke actual main function.
And this wrapper provides workaround for oozie's System.exit problem.
(https://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a3.2.7_Java_Action)
In caller of oozie, System.exit is converted to exception.
The application can call System.exit multiple times.
This wrapper stores the argument of System.exit called in first.
And return stored value if System.exit is called multiple times.

55
edp-adapt-for-oozie/pom.xml
View File

@ -1,55 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.openstack.sahara.edp</groupId>
<artifactId>edp-main-wrapper</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>EDP Java Action Main Wrapper for oozie</name>
<packaging>jar</packaging>
<properties>
<file.encoding>UTF-8</file.encoding>
<downloadSources>true</downloadSources>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<configLocation>file://${basedir}/../hadoop-swiftfs/checkstyle.xml</configLocation>
<failOnViolation>false</failOnViolation>
<format>xml</format>
<format>html</format>
</configuration>
</plugin>
</plugins>
</build>
</project>

91
edp-adapt-for-oozie/src/main/java/org/openstack/sahara/edp/MainWrapper.java
View File

@ -1,91 +0,0 @@
package org.openstack.sahara.edp;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.security.Permission;
import java.util.Arrays;
public class MainWrapper {
public static void main(String[] args) throws Throwable {
// Load oozie configuration file
String actionConf = System.getProperty("oozie.action.conf.xml");
if (actionConf != null) {
Class<?> configClass
= Class.forName("org.apache.hadoop.conf.Configuration");
Method method = configClass.getMethod("addDefaultResource", String.class);
method.invoke(null, "action.xml");
}
SecurityManager originalSecurityManager = System.getSecurityManager();
WrapperSecurityManager newSecurityManager
= new WrapperSecurityManager(originalSecurityManager);
System.setSecurityManager(newSecurityManager);
Class<?> mainClass = Class.forName(args[0]);
Method mainMethod = mainClass.getMethod("main", String[].class);
String[] newArgs = Arrays.copyOfRange(args, 1, args.length);
Throwable exception = null;
try {
mainMethod.invoke(null, (Object) newArgs);
} catch (InvocationTargetException e) {
if (!newSecurityManager.getExitInvoked()) {
exception = e.getTargetException();
}
}
System.setSecurityManager(originalSecurityManager);
if (exception != null) {
throw exception;
}
if (newSecurityManager.getExitInvoked()) {
System.exit(newSecurityManager.getExitCode());
}
}
static class WrapperSecurityManager extends SecurityManager {
private static boolean exitInvoked = false;
private static int firstExitCode;
private SecurityManager securityManager;
public WrapperSecurityManager(SecurityManager securityManager) {
this.securityManager = securityManager;
}
@Override
public void checkPermission(Permission perm, Object context) {
if (securityManager != null) {
// check everything with the original SecurityManager
securityManager.checkPermission(perm, context);
}
}
@Override
public void checkPermission(Permission perm) {
if (securityManager != null) {
// check everything with the original SecurityManager
securityManager.checkPermission(perm);
}
}
@Override
public void checkExit(int status) throws SecurityException {
if (!exitInvoked) {
// save first System.exit status code
exitInvoked = true;
firstExitCode = status;
}
throw new SecurityException("Intercepted System.exit(" + status + ")");
}
public static boolean getExitInvoked() {
return exitInvoked;
}
public static int getExitCode() {
return firstExitCode;
}
}
}

14
edp-adapt-for-spark/README.rst
View File

@ -1,14 +0,0 @@
===========================================
Sources for main function wrapper for Spark
===========================================
The Hadoop configuration for a Spark job must be modified if
the Spark job is going to access Swift paths. Specifically,
the Hadoop configuration must contain values that allow
the job to authenticate to the Swift service.
This wrapper adds a specified xml file to the default Hadoop
Configuration resource list and then calls the specified
main class. Any necessary Hadoop configuration values can
be added to the xml file. This allows the main class to
be run and access Swift paths without alteration.

54
edp-adapt-for-spark/pom.xml
View File

@ -1,54 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.openstack.sahara.edp</groupId>
<artifactId>edp-spark-wrapper</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>EDP Wrapper for Spark</name>
<packaging>jar</packaging>
<properties>
<file.encoding>UTF-8</file.encoding>
<downloadSources>true</downloadSources>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<configLocation>file://${basedir}/../hadoop-swiftfs/checkstyle.xml</configLocation>
<failOnViolation>false</failOnViolation>
<format>xml</format>
<format>html</format>
</configuration>
</plugin>
</plugins>
</build>
</project>

22
edp-adapt-for-spark/src/main/java/org/openstack/sahara/edp/SparkWrapper.java
View File

@ -1,22 +0,0 @@
package org.openstack.sahara.edp;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.security.Permission;
import java.util.Arrays;
public class SparkWrapper {
public static void main(String[] args) throws Throwable {
Class<?> configClass
= Class.forName("org.apache.hadoop.conf.Configuration");
Method method = configClass.getMethod("addDefaultResource", String.class);
method.invoke(null, args[0]);
Class<?> mainClass = Class.forName(args[1]);
Method mainMethod = mainClass.getMethod("main", String[].class);
String[] newArgs = Arrays.copyOfRange(args, 2, args.length);
mainMethod.invoke(null, (Object) newArgs);
}
}

5
edp-examples/README.rst
View File

@ -1,5 +0,0 @@
EDP Examples
============
All files from this directory have been moved to new
sahara-tests repository: https://opendev.org/openstack/sahara-tests

17
hadoop-swiftfs/README.rst
View File

@ -1,17 +0,0 @@
======================================================
Sources for Swift filesystem implementation for Hadoop
======================================================
These sources were originally published at
https://issues.apache.org/jira/secure/attachment/12583703/HADOOP-8545-033.patch
The sources were obtained by running "patch" command. All the files related to
Hadoop-common were skipped during patching.
Changes were made after patching:
* pom.xml was updated to use hadoop-core 1.1.2 dependency and adds hadoop2
profile
* removed dependency on 2.x hadoop in code (@Override and isDirectory()
-> isDir())
* removed Hadoop 2.X tests
There are no unit-tests, only integration.

189
hadoop-swiftfs/checkstyle.xml
View File

@ -1,189 +0,0 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.2//EN"
"http://www.puppycrawl.com/dtds/configuration_1_2.dtd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
Checkstyle configuration for Hadoop that is based on the sun_checks.xml file
that is bundled with Checkstyle and includes checks for:
- the Java Language Specification at
http://java.sun.com/docs/books/jls/second_edition/html/index.html
- the Sun Code Conventions at http://java.sun.com/docs/codeconv/
- the Javadoc guidelines at
http://java.sun.com/j2se/javadoc/writingdoccomments/index.html
- the JDK Api documentation http://java.sun.com/j2se/docs/api/index.html
- some best practices
Checkstyle is very configurable. Be sure to read the documentation at
http://checkstyle.sf.net (or in your downloaded distribution).
Most Checks are configurable, be sure to consult the documentation.
To completely disable a check, just comment it out or delete it from the file.
Finally, it is worth reading the documentation.
-->
<module name="Checker">
<!-- Checks that a package.html file exists for each package. -->
<!-- See http://checkstyle.sf.net/config_javadoc.html#PackageHtml -->
<module name="JavadocPackage"/>
<!-- Checks whether files end with a new line. -->
<!-- See http://checkstyle.sf.net/config_misc.html#NewlineAtEndOfFile -->
<!-- module name="NewlineAtEndOfFile"/-->
<!-- Checks that property files contain the same keys. -->
<!-- See http://checkstyle.sf.net/config_misc.html#Translation -->
<module name="Translation"/>
<module name="FileLength"/>
<module name="FileTabCharacter"/>
<module name="TreeWalker">
<!-- Checks for Javadoc comments. -->
<!-- See http://checkstyle.sf.net/config_javadoc.html -->
<module name="JavadocType">
<property name="scope" value="public"/>
<property name="allowMissingParamTags" value="true"/>
</module>
<module name="JavadocStyle"/>
<!-- Checks for Naming Conventions. -->
<!-- See http://checkstyle.sf.net/config_naming.html -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName"/>
<module name="LocalVariableName"/>
<module name="MemberName"/>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
<!-- Checks for Headers -->
<!-- See http://checkstyle.sf.net/config_header.html -->
<!-- <module name="Header"> -->
<!-- The follow property value demonstrates the ability -->
<!-- to have access to ANT properties. In this case it uses -->
<!-- the ${basedir} property to allow Checkstyle to be run -->
<!-- from any directory within a project. See property -->
<!-- expansion, -->
<!-- http://checkstyle.sf.net/config.html#properties -->
<!-- <property -->
<!-- name="headerFile" -->
<!-- value="${basedir}/java.header"/> -->
<!-- </module> -->
<!-- Following interprets the header file as regular expressions. -->
<!-- <module name="RegexpHeader"/> -->
<!-- Checks for imports -->
<!-- See http://checkstyle.sf.net/config_import.html -->
<module name="IllegalImport"/> <!-- defaults to sun.* packages -->
<module name="RedundantImport"/>
<module name="UnusedImports"/>
<!-- Checks for Size Violations. -->
<!-- See http://checkstyle.sf.net/config_sizes.html -->
<module name="LineLength"/>
<module name="MethodLength"/>
<module name="ParameterNumber"/>
<!-- Checks for whitespace -->
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
<module name="EmptyForIteratorPad"/>
<module name="MethodParamPad"/>
<module name="NoWhitespaceAfter"/>
<module name="NoWhitespaceBefore"/>
<module name="ParenPad"/>
<module name="TypecastParenPad"/>
<module name="WhitespaceAfter">
<property name="tokens" value="COMMA, SEMI"/>
</module>
<module name="WhitespaceAround">
<property name="tokens" value="ASSIGN"/>
</module>
<!-- Modifier Checks -->
<!-- See http://checkstyle.sf.net/config_modifiers.html -->
<module name="ModifierOrder"/>
<module name="RedundantModifier"/>
<!-- Checks for blocks. You know, those {}'s -->
<!-- See http://checkstyle.sf.net/config_blocks.html -->
<module name="AvoidNestedBlocks"/>
<module name="EmptyBlock"/>
<module name="LeftCurly"/>
<module name="NeedBraces"/>
<module name="RightCurly"/>
<!-- Checks for common coding problems -->
<!-- See http://checkstyle.sf.net/config_coding.html -->
<!-- module name="AvoidInlineConditionals"/-->
<!--<module name="DoubleCheckedLocking"/>-->
<module name="EmptyStatement"/>
<module name="EqualsHashCode"/>
<module name="HiddenField">
<property name="ignoreConstructorParameter" value="true"/>
</module>
<module name="IllegalInstantiation"/>
<module name="InnerAssignment"/>
<module name="MissingSwitchDefault"/>
<module name="RedundantThrows"/>
<module name="SimplifyBooleanExpression"/>
<module name="SimplifyBooleanReturn"/>
<!-- Checks for class design -->
<!-- See http://checkstyle.sf.net/config_design.html -->
<module name="FinalClass"/>
<module name="HideUtilityClassConstructor"/>
<module name="InterfaceIsType"/>
<module name="VisibilityModifier"/>
<!-- Miscellaneous other checks. -->
<!-- See http://checkstyle.sf.net/config_misc.html -->
<module name="ArrayTypeStyle"/>
<module name="Indentation">
<property name="basicOffset" value="2" />
<property name="caseIndent" value="0" />
</module>
<module name="TodoComment"/>
<module name="UpperEll"/>
</module>
</module>

194
hadoop-swiftfs/pom.xml
View File

@ -1,194 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-openstack</artifactId>
<version>3.0.0-SNAPSHOT</version>
<name>Apache Hadoop OpenStack support</name>
<description>
This module contains code to support integration with OpenStack.
Currently this consists of a filesystem client to read data from
and write data to an OpenStack Swift object store.
</description>
<packaging>jar</packaging>
<properties>
<targetJavaVersion>1.6</targetJavaVersion>
<sourceJavaVersion>1.6</sourceJavaVersion>
<file.encoding>UTF-8</file.encoding>
<downloadSources>true</downloadSources>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hadoop.artifactid>hadoop-core</hadoop.artifactid>
<hadoop.version>1.2.1</hadoop.version>
</properties>
<profiles>
<profile>
<id>tests-off</id>
<activation>
<file>
<missing>src/test/resources/auth-keys.xml</missing>
</file>
</activation>
<properties>
<maven.test.skip>true</maven.test.skip>
</properties>
</profile>
<profile>
<id>tests-on</id>
<activation>
<file>
<exists>src/test/resources/auth-keys.xml</exists>
</file>
</activation>
<properties>
<maven.test.skip>false</maven.test.skip>
</properties>
</profile>
<profile>
<id>hadoop2</id>
<properties>
<hadoop.artifactid>hadoop-common</hadoop.artifactid>
<hadoop.version>2.4.1</hadoop.version>
</properties>
</profile>
<profile>
<id>hadoop3</id>
<properties>
<hadoop.artifactid>hadoop-common</hadoop.artifactid>
<hadoop.version>3.0.1</hadoop.version>
<targetJavaVersion>1.8</targetJavaVersion>
<sourceJavaVersion>1.8</sourceJavaVersion>
</properties>
</profile>
</profiles>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.maven.doxia</groupId>
<artifactId>doxia-module-markdown</artifactId>
<version>1.3</version>
</dependency>
</dependencies>
<configuration>
<inputEncoding>UTF-8</inputEncoding>
<outputEncoding>UTF-8</outputEncoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-project-info-reports-plugin</artifactId>
<configuration>
<dependencyDetailsEnabled>false</dependencyDetailsEnabled>
<dependencyLocationsEnabled>false</dependencyLocationsEnabled>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${sourceJavaVersion}</source>
<target>${targetJavaVersion}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<configLocation>file://${basedir}/checkstyle.xml</configLocation>
<failOnViolation>false</failOnViolation>
<format>xml</format>
<format>html</format>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop.artifactid}</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.12</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.12</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.2.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
<scope>compile</scope>
</dependency>
<!-- Used for loading test resources and converting a File to byte[] -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
<scope>compile</scope>
</dependency>
<!-- Used for mocking dependencies -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.8.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>test</scope>
<version>14.0.1</version>
</dependency>
</dependencies>
</project>

66
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/ApiKeyAuthenticationRequest.java
View File

@ -1,66 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
import org.codehaus.jackson.annotate.JsonProperty;
/**
* Class that represents authentication request to Openstack Keystone.
* Contains basic authentication information.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS
*/
public class ApiKeyAuthenticationRequest extends AuthenticationRequestV2 {
/**
* Credentials for login
*/
private ApiKeyCredentials apiKeyCredentials;
/**
* API key auth
* @param tenantName tenant
* @param apiKeyCredentials credentials
*/
public ApiKeyAuthenticationRequest(String tenantName, ApiKeyCredentials apiKeyCredentials) {
this.tenantName = tenantName;
this.apiKeyCredentials = apiKeyCredentials;
}
/**
* @return credentials for login into Keystone
*/
@JsonProperty("RAX-KSKEY:apiKeyCredentials")
public ApiKeyCredentials getApiKeyCredentials() {
return apiKeyCredentials;
}
/**
* @param apiKeyCredentials credentials for login into Keystone
*/
public void setApiKeyCredentials(ApiKeyCredentials apiKeyCredentials) {
this.apiKeyCredentials = apiKeyCredentials;
}
@Override
public String toString() {
return "Auth as " +
"tenant '" + tenantName + "' "
+ apiKeyCredentials;
}
}

87
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/ApiKeyCredentials.java
View File

@ -1,87 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
/**
* Describes credentials to log in Swift using Keystone authentication.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class ApiKeyCredentials {
/**
* user login
*/
private String username;
/**
* user password
*/
private String apikey;
/**
* default constructor
*/
public ApiKeyCredentials() {
}
/**
* @param username user login
* @param apikey user api key
*/
public ApiKeyCredentials(String username, String apikey) {
this.username = username;
this.apikey = apikey;
}
/**
* @return user api key
*/
public String getApiKey() {
return apikey;
}
/**
* @param apikey user api key
*/
public void setApiKey(String apikey) {
this.apikey = apikey;
}
/**
* @return login
*/
public String getUsername() {
return username;
}
/**
* @param username login
*/
public void setUsername(String username) {
this.username = username;
}
@Override
public String toString() {
return "user " +
"'" + username + '\'' +
" with key of length " + ((apikey == null) ? 0 : apikey.length());
}
}

36
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/AuthenticationRequest.java
View File

@ -1,36 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
/**
* Class that represents authentication request to Openstack Keystone.
* Contains basic authentication information.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class AuthenticationRequest {
public AuthenticationRequest() {
}
@Override
public String toString() {
return "AuthenticationRequest";
}
}

57
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/AuthenticationRequestV2.java
View File

@ -1,57 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
/**
* Class that represents authentication request to Openstack Keystone.
* Contains basic authentication information.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class AuthenticationRequestV2 extends AuthenticationRequest {
/**
* tenant name
*/
protected String tenantName;
public AuthenticationRequestV2() {
}
/**
* @return tenant name for Keystone authorization
*/
public String getTenantName() {
return tenantName;
}
/**
* @param tenantName tenant name for authorization
*/
public void setTenantName(String tenantName) {
this.tenantName = tenantName;
}
@Override
public String toString() {
return "AuthenticationRequestV2{" +
"tenantName='" + tenantName + '\'' +
'}';
}
}

32
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/AuthenticationRequestV3.java
View File

@ -1,32 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
/**
* Class that represents authentication request to Openstack Keystone.
* Contains basic authentication information.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class AuthenticationRequestV3 extends AuthenticationRequest {
public AuthenticationRequestV3() {
}
}

59
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/AuthenticationRequestWrapper.java
View File

@ -1,59 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
/**
* This class is used for correct hierarchy mapping of
* Keystone authentication model and java code.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class AuthenticationRequestWrapper {
/**
* authentication request
*/
private AuthenticationRequest auth;
/**
* default constructor used for json parsing
*/
public AuthenticationRequestWrapper() {
}
/**
* @param auth authentication requests
*/
public AuthenticationRequestWrapper(AuthenticationRequest auth) {
this.auth = auth;
}
/**
* @return authentication request
*/
public AuthenticationRequest getAuth() {
return auth;
}
/**
* @param auth authentication request
*/
public void setAuth(AuthenticationRequest auth) {
this.auth = auth;
}
}

69
hadoop-swiftfs/src/main/java/org/apache/hadoop/fs/swift/auth/AuthenticationResponse.java
View File

@ -1,69 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.fs.swift.auth;
import org.apache.hadoop.fs.swift.auth.entities.AccessToken;
import org.apache.hadoop.fs.swift.auth.entities.Catalog;
import org.apache.hadoop.fs.swift.auth.entities.User;
import java.util.List;
/**
* Response from KeyStone deserialized into AuthenticationResponse class.
* THIS FILE IS MAPPED BY JACKSON TO AND FROM JSON.
* DO NOT RENAME OR MODIFY FIELDS AND THEIR ACCESSORS.
*/
public class AuthenticationResponse {
private Object metadata;
private List<Catalog> serviceCatalog;
private User user;
private AccessToken token;
public Object getMetadata() {
return metadata;
}
public void setMetadata(Object metadata) {
this.metadata = metadata;
}