4a781a7f86
yarn drives package and dependency management. webpack handles bundling, minification and transpiling down to browser-acceptable javascript but allows for more modern javascript like import statements. There are some really neat things in the webpack dev server. CSS changes, for instance, get applied immediately without a refresh. Other things, like the jquery plugin do need a refresh, but it's handled just on a file changing. As a followup, we can also consider turning the majority of the status page into a webpack library that other people can depend on as a mechanism for direct use. Things like that haven't been touched because allowing folks to poke at the existing known status page without too many changes using the tools seems like a good way for people to learn/understand the stack. Move things so that the built content gets put into zuul/web/static so that the built-in static serving from zuul-web will/can serve the files. Update MANIFEST.in so that if npm run build:dist is run before the python setup.py sdist, the built html/javascript content will be included in the source tarball. Add a pbr hook so that if yarn is installed, javascript content will be built before the tarball. Add a zuul job with a success url that contains a source_url pointing to the live v3 data. This adds a framework for verifying that we can serve the web app urls and their dependencies for all of the various ways we want to support folks hosting zuul-web. It includes a very simple reverse proxy server for approximating what we do in openstack to "white label" the Zuul service -- that is, hide the multitenancy aspect and present the single tenant at the site root. We can run similar tests without the proxy to ensure the default, multi-tenant view works as well. Add babel transpiling enabling use of ES6 features ECMAScript6 has a bunch of nice things, like block scoped variables, const, template strings and classes. Babel is a javascript transpiler which webpack can use to allow us to write using modern javascript but the resulting code to still work on older browsers. Use the babel-plugin-angularjs-annotate so that angular's dependency injection doesn't get borked by babel's transpiling things (which causes variables to otherwise be renamed in a way that causes angular to not find them) While we're at it, replace our use of var with let (let is the new block-scoped version of var) and toss in some use of const and template strings for good measure. Add StandardJS eslint config for linting JavaScript Standard Style is a code style similar to pep8/flake8. It's being added here not because of the pep8 part, but because the pyflakes equivalent can catch real errors. This uses the babel-eslint parser since we're using Babel to transpile already. This auto-formats the existing code with: npm run format Rather than using StandardJS directly through the 'standard' package, use the standardjs eslint plugin so that we can ignore the camelCase rule (and any other rule that might emerge in the future) Many of under_score/camelCase were fixed in a previous version of the patch. Since the prevailing zuul style is camelCase methods anyway, those fixes were left. That warning has now been disabled. Other things, such as == vs. === and ensuring template strings are in backticks are fixed. Ignore indentation errors for now - we'll fix them at the end of this stack and then remove the exclusion. Add a 'format' npm run target that will run the eslint command with --fix for ease of fixing reported issues. Add a 'lint' npm run target and a 'lint' environment that runs with linting turned to errors. The next patch makes the lint environment more broadly useful. When we run lint, also run the BundleAnalyzerPlugin and set the success-url to the report. Add an angular controller for status and stream page Wrap the status and stream page construction with an angular controller so that all the javascripts can be bundled in a single file. Building the files locally is wonderful and all, but what we really want is to make a tarball that has the built code so that it can be deployed. Put it in the root source dir so that it can be used with the zuul fetch-javascript-tarball role. Also, replace the custom npm job with the new build-javascript-content job which naturally grabs the content we want. Make a 'main.js' file that imports the other three so that we just have a single bundle. Then, add a 'vendor' entry in the common webpack file and use the CommonsChunkPlugin to extract dependencies into their own bundle. A second CommonsChunkPlugin entry pulls out a little bit of metadata that would otherwise cause the main and vendor chunks to change even with no source change. Then add chunkhash into the filename. This way the files themselves can be aggressively cached. This all follows recommendations from https://webpack.js.org/guides/caching/ https://webpack.js.org/guides/code-splitting/ and https://webpack.js.org/guides/output-management/ Change-Id: I2e1230783fe57f1bc3b7818460463df1e659936b Co-Authored-By: Tristan Cacqueray <tdecacqu@redhat.com> Co-Authored-By: James E. Blair <jeblair@redhat.com> |
||
|---|---|---|
| doc | ||
| etc | ||
| playbooks | ||
| tests | ||
| tools | ||
| web | ||
| zuul | ||
| .babelrc | ||
| .eslintrc | ||
| .gitignore | ||
| .gitreview | ||
| .mailmap | ||
| .testr.conf | ||
| .zuul.yaml | ||
| LICENSE | ||
| MANIFEST.in | ||
| NEWS.rst | ||
| README.rst | ||
| TESTING.rst | ||
| bindep.txt | ||
| package.json | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
| webpack.config.js | ||
| yarn.lock | ||
README.rst
Zuul
Zuul is a project gating system developed for the OpenStack Project.
We are currently engaged in a significant development effort in preparation for the third major version of Zuul. We call this effort Zuul v3 and it is described in more detail below.
The latest documentation for Zuul v3 is published at: https://docs.openstack.org/infra/zuul/
If you are looking for the Edge routing service named Zuul that is related to Netflix, it can be found here: https://github.com/Netflix/zuul
If you are looking for the Javascript testing tool named Zuul, it can be found here: https://github.com/defunctzombie/zuul
Contributing
We are currently engaged in a significant development effort in preparation for the third major version of Zuul. We call this effort Zuul v3.
To browse the latest code, see: https://git.openstack.org/cgit/openstack-infra/zuul/tree/ To clone the latest code, use git clone git://git.openstack.org/openstack-infra/zuul
Bugs are handled at: https://storyboard.openstack.org/#!/project/679
Code reviews are, as you might expect, handled by gerrit at https://review.openstack.org
Use git review to submit patches (after creating a Gerrit account that links to your launchpad account). Example:
# Do your commits
$ git review
# Enter your username if promptedZuul v3
The Zuul v3 effort involves significant changes to Zuul, and its companion program, Nodepool. The intent is for Zuul to become more generally useful outside of the OpenStack community. This is the best way to get started with this effort:
Read the Zuul v3 spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
We use specification documents like this to describe large efforts where we want to make sure that all the participants are in agreement about what will happen and generally how before starting development. These specs should contain enough information for people to evaluate the proposal generally, and sometimes include specific details that need to be agreed upon in advance. They are living documents which can change as work gets underway. However, every change or detail does not need to be reflected in the spec --most work is simply done with patches (and revised if necessary in code review).
Read the Nodepool build-workers spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html
Review any proposed updates to these specs: https://review.openstack.org/#/q/status:open+project:openstack-infra/infra-specs+topic:zuulv3
Some of the information in the specs may be effectively superceded by changes here, which are still undergoing review.
Read developer documentation on the internal data model and testing: http://docs.openstack.org/infra/zuul/developer.html
The general philosophy for Zuul tests is to perform functional testing of either the individual component or the entire end-to-end system with external systems (such as Gerrit) replaced with fakes. Before adding additional unit tests with a narrower focus, consider whether they add value to this system or are merely duplicative of functional tests.
Review open changes: https://review.openstack.org/#/q/status:open
We find that the most valuable code reviews are ones that spot problems with the proposed change, or raise questions about how that might affect other systems or subsequent work. It is also a great way to stay involved as a team in work performed by others (for instance, by observing and asking questions about development while it is in progress). We try not to sweat the small things and don't worry too much about style suggestions or other nitpicky things (unless they are relevant -- for instance, a -1 vote on a change that introduces a yaml change out of character with existing conventions is useful because it makes the system more user-friendly; a -1 vote on a change which uses a sub-optimal line breaking strategy is probably not the best use of anyone's time).
Join #zuul on Freenode. Let others (especially jeblair who is trying to coordinate and prioritize work) know what you would like to work on.
Check storyboard for status of current work items: https://storyboard.openstack.org/#!/board/41
Work items tagged with
low-hanging-fruitare tasks that have been identified as not requiring an expansive knowledge of the system. They may still require either some knowledge or investigation into a specific area, but should be suitable for a developer who is becoming acquainted with the system. Those items can be found at: https://storyboard.openstack.org/#!/story/list?tags=low-hanging-fruit&tags=zuulv3
Once you are up to speed on those items, it will be helpful to know the following:
Zuul v3 includes some substantial changes to Zuul, and in order to implement them quickly and simultaneously, we temporarily disabled most of the test suite. That test suite still has relevance, but tests are likely to need updating individually, with reasons ranging from something simple such as a test-framework method changing its name, to more substantial issues, such as a feature being removed as part of the v3 work. Each test will need to be evaluated individually. Feel free to, at any time, claim a test name in this story and work on re-enabling it: https://storyboard.openstack.org/#!/story/2000773
Because of the importance of external systems, as well as the number of internal Zuul components, actually running Zuul in a development mode quickly becomes unweildy (imagine uploading changes to Gerrit repeatedly while altering Zuul source code). Instead, the best way to develop with Zuul is in fact to write a functional test. Construct a test to fully simulate the series of events you want to see, then run it in the foreground. For example:
.tox/py27/bin/python -m testtools.run tests.unit.test_scheduler.TestScheduler.test_jobs_executedSee TESTING.rst for more information.
There are many occasions, when working on sweeping changes to Zuul v3, we left notes for future work items in the code marked with "TODOv3". These represent potentially serious missing functionality or other issues which must be resolved before an initial v3 release (unlike a more conventional TODO note, these really can not be left indefinitely). These present an opportunity to identify work items not otherwise tracked. The names associated with TODO or TODOv3 items do not mean that only that person can address them -- they simply reflect who to ask to explain the item in more detail if it is too cryptic. In your own work, feel free to leave TODOv3 notes if a change would otherwise become too large or unweildy.
Python Version Support
Zuul v3 requires Python 3. It does not support Python 2.
As Ansible is used for the execution of jobs, it's important to note
that while Ansible does support Python 3, not all of Ansible's modules
do. Zuul currently sets ansible_python_interpreter to
python2 so that remote content will be executed with Python2.
Roadmap
- Begin using Zuul v3 to run jobs for Zuul itself
- Implement a shim to translate Zuul v2 demand into Nodepool Zookeeper launcher requests
- Begin using Zookeeper based Nodepool launchers with Zuul v2.5 in OpenStack Infra
- Move OpenStack Infra to use Zuul v3
- Implement Github support
- Begin using Zuul v3 to run tests on Ansible repos
- Implement support in Nodepool for non-OpenStack clouds
- Add native container support to Zuul / Nodepool