zware/awx
1
0
Fork 0
mirror of https://github.com/ZwareBear/awx.git synced 2026-03-20 07:43:35 -05:00
Code Issues Packages Projects Releases Wiki Activity

General upgrade of dependencies

Browse Source 
Update licenses for new versions after dependency upgrades

pin pycurl to version that does not break on install

implement new workflow for py2/3 requirements management

require twisted tls extras, resolve service-identity version

Upgrade celery to resolve importlib DeprecationWarning

use flags to resolve the unsafe and cache problems
This commit is contained in:
AlanCoding
2019-04-11 10:49:00 -04:00
parent f174902bb2
commit 2bec5ddb41
13 changed files with 149 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
requirements/README.md
View File

@@ -1,23 +1,46 @@
The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tower_tools container.
The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tools_awx container.
NOTE: before running `pip-compile`, please copy-paste contents in `requirements/requirements_git.txt` to the top of `requirements/requirements.in` and prepend each copied line with `-e `. Later after `requirements.txt` is generated, don't forget to remove all `git+https://github.com...`-like lines from both `requirements.txt` and `requirements.in` (repeat for `requirements_ansible_git.txt` and `requirements_ansible.in`)
At the end of `requirements/requirements.in`, pip and setuptools need to have their versions pinned.
Run these commands from the root of the awx repo. This will produce python 3 requirements files.
```
python3 -m venvpip install pip-tools /buildit
python3 -m venv /buildit
source /buildit/bin/activate
pip install pip-tools
pip install pip --upgrade
pip-compile requirements/requirements.in > requirements/requirements.txt
pip-compile requirements/requirements_ansible.in > requirements/requirements_ansible.txt
pip-compile -U -r --allow-unsafe --output-file requirements/requirements.txt requirements/requirements.in
pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible_py3.txt requirements/requirements_ansible.in
```
## Known Issues
The Ansible venv requirements file needs to start with the python 2 version
as a base. Then we can run the tool again to get the python 3 version.
Consult the output of the `diff` command and add a conditional switch in those cases.
* Remove the `-e` from packages of the form `-e git+https://github.com...` in the generated `.txt`. Failure to do so will result in a "bad" RPM and DEB due to the `pip install` laying down a symbolic link with an absolute path from the virtualenv to the git repository that will differ from when the RPM and DEB are build to when the RPM and DEB are installed on a machine. By removing the `-e` the symbolic egg link will not be created and all is well.
```
virtualenv -p python2 /buildit_py2
source /buildit_py2/bin/activate
pip install pip-tools
pip install pip --upgrade
* As of `pip-tools` `1.8.1` `pip-compile` does not resolve packages specified using a git url. Thus, dependencies for things like `dm.xmlsec.binding` do not get resolved and output to `requirements.txt`. This means that:
* can't use `pip install --no-deps` because other deps WILL be sucked in
* all dependencies are NOT captured in our `.txt` files. This means you can't rely on the `.txt` when gathering licenses.
pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
diff requirements/requirements_ansible_py3.txt requirements/requirements_ansible.txt
rm requirements/requirements_ansible_py3.txt
```
Python 3 exceptions should be added to relevant `requirements_ansible.txt` lines
after version numbers with the syntax of `; python_version < '3'`.
## Licenses and Source Files
If any library has a change to its license with the upgrade, then the license for that library
inside of `docs/licenses` needs to be updated.
For libraries that have source distribution requirements (LGPL as an example),
a tarball of the library is kept along with the license.
To download the PyPI tarball, you can run this command:
```
pip download <pypi library name> -d docs/licenses/ --no-binary :all: --no-deps
```
Make sure to delete the old tarball if it is an upgrade.