Commit Graph

901 Commits

Author SHA1 Message Date
Ryan Petrello
46795cc71e fix a bug that caused parent_uuid not to be persisted 2019-05-01 10:13:59 -04:00
Ryan Petrello
eb0463890c add a loud warning about perf regressions to tasks.py 2019-04-22 10:26:58 -04:00
Vismay Golwala
5d570a017a Validate virtual environment while running a job/inventory update
Currently we only check the custom virtual environment path when
it's created. However, to tackle with the case when the venv might
have been changed/deleted afterward, we need to validate it at
run-time too.

Signed-off-by: Vismay Golwala <vgolwala@redhat.com>
2019-04-18 08:09:49 -04:00
chris meyers
84c09a19d1 instantiate dispatcher once per job run
* Instantiating the callback dispatch queue on each job event callback
is expensive. Instead, instantiate it only once. Note, we do not need to
instantiate the callback queue in the iso case so we do not.
2019-04-17 14:03:00 -04:00
Ryan Petrello
387682ed8d if runner crashes, attempt to record why
this attempts to surface the underlying runner exception for tracebacks
like this one:

FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/awx_41_93gtgv25/artifacts/41/status'
2019-04-15 13:17:45 -04:00
softwarefactory-project-zuul[bot]
e3dfc6c796 Merge pull request #3596 from jbradberry/capture-isolated-command
Updated IsolatedManager to take a callback that captures the remote command

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-04-05 17:15:11 +00:00
Ryan Petrello
81fe923577 don't write playbook stdout to sys.stdout (it's duplicated in log files)
this instructs runner to _not_ write to stdout when we invoke
runner.interface.run(); AWX consumes/ingests this strictly as events
2019-04-05 11:20:34 -04:00
Jeff Bradberry
c6643946c5 Capture the redacted credential env vars separately
and then make use of them specifically to make safe the env vars
coming back from an isolated node.  This will allow us to capture the
safed versions of custom credential values, but without potentially
clobbering normal env var values that vary between the controller and
the node.
2019-04-04 15:22:27 -04:00
softwarefactory-project-zuul[bot]
5debfdf5e8 Merge pull request #3545 from chrismeyersfsu/fix-tower_header_on_insights_playbook_fetches
include tower header to insights on plan fetch

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-04-04 15:51:06 +00:00
Jeff Bradberry
b0f6d2214c Fix a typo: there is no method called check_callback on BaseTask 2019-04-03 14:57:02 -04:00
Jeff Bradberry
4364e00117 Do the env vars redaction for isolated nodes on this side 2019-04-03 14:34:09 -04:00
Jeff Bradberry
32286a9d49 Change the artifact to also capture the actual envvars data 2019-04-02 17:10:26 -04:00
Jeff Bradberry
cac48e7cfb Updated IsolatedManager to take a callback that captures the remote command 2019-04-02 15:40:56 -04:00
Ryan Petrello
81a509424a prefetch related source credentials in tasks.py 2019-04-02 11:21:33 -04:00
Ryan Petrello
69368d874e move path parameterization to the CredentialInputSource model 2019-04-02 11:20:36 -04:00
Ryan Petrello
7a43f00a5d add support for HashiCorp signed SSH certificates 2019-04-02 11:20:29 -04:00
chris meyers
71fcb1a82c process host facts for iso runs
* Move isolated clean to our final run hook
* ISO and non-iso code path now share the post-fact-processing code
2019-03-29 16:16:22 -04:00
Ryan Petrello
563a0cc2a4 move awx.main.expect to awx.main.isolated 2019-03-29 12:14:40 -04:00
Ryan Petrello
d663d397f8 clean up some isolated adhoc code
if it's adhoc, there's not project directory to copy
2019-03-29 10:31:56 -04:00
softwarefactory-project-zuul[bot]
2ab290ff2d Merge pull request #3534 from ryanpetrello/iso-pexpect-cleanup
replace our usage of pexpect in IsolatedManager with ansible-runner

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-03-29 13:17:05 +00:00
softwarefactory-project-zuul[bot]
ef88507d23 Merge pull request #3543 from AlanCoding/do_not_transform
Set Ansible name transform setting for built-in sources

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-03-28 23:47:56 +00:00
Ryan Petrello
9160d91278 clean up old usage of idle_timeout
cmeyers and I looked at this and can't tell where/why/how you'd actually
set this setting - it looks like really old ~2014-2015 Tower history
that probably isn't actually in use
2019-03-28 17:31:32 -04:00
Ryan Petrello
dd81f59d9a set PYTHONPATH properly for isolated usage of runner 2019-03-28 12:40:21 -04:00
Ryan Petrello
99478f5d25 replace our usage of pexpect in IsolatedManager with ansible-runner 2019-03-28 12:40:21 -04:00
chris meyers
926d6a6525 include tower header to insights on plan fetch
* We include a special header value in the user agent when tower proxies
requests to get per-host rules.
* This extends that header logic to when we fetch plans (playbooks)
2019-03-28 10:56:56 -04:00
AlanCoding
e84642b4a1 Set Ansible name transform setting for built-in sources
This is the current default but will change in the future
for these sources, we do not want behavior change
2019-03-28 10:37:36 -04:00
Christian Adams
ae9377e0e4 add basic job & JT analytics 2019-03-27 19:53:00 -04:00
Ryan Petrello
c586fa9821 add a minimal framework for generating analytics/metrics
annotate queries & add license analytics
2019-03-27 19:53:00 -04:00
softwarefactory-project-zuul[bot]
7b4c63037a Merge pull request #3523 from ryanpetrello/iso-cancel
properly handle isolated cancellation

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-03-27 22:44:10 +00:00
Ryan Petrello
a983d4bc1f properly handle isolated cancellation 2019-03-27 11:44:26 -04:00
chris meyers
2329079326 runner expects process isolation flags in settings
* Towards the goal of converging the iso code path w/ the non-iso code
path. More process isolation control flags into settings.
2019-03-27 11:08:41 -04:00
AlanCoding
d39b3b3165 Remove compatibility_mode field, simplify jinja2 syntax
fix minor bug where credential not shown in API
2019-03-26 10:29:39 -04:00
AlanCoding
19ad7d3983 Inventory plugins data tweaks and finalization
Disable use of azure_rm inventory plugin
Disable use of ec2 inventory plugin
due to compatibility issues that are unresolved

Fix conflicts with ansible runner integration

Add additional content enabled by Ansible core changes
2019-03-26 10:29:39 -04:00
AlanCoding
cd7e358b73 Inventory plugins transition dev finishing work
Bump keystone auth to resolve problem with openstack script

Clarify code path, routing to template vs. managed injector
  behavior is also now reflected in test data files

Refactor test data layout for inventory injector logic

Add developer docs for inventory plugins transition

Memoize only get_ansible_version with no parameters

Make inventory plugin injector enablement a separate
  concept from the initial_version
  switch tests to look for plugin_name as well

Add plugin injectors for tower and foreman.

Add jinja2 native types compat feature

move tower source license compare logic to management command

introduce inventory source compat mode

pin jinja2 for native Ansible types

Add parent group keys, and additional translations

manual dash sanitization for un-region-like ec2 groups

nest zones under regions using Ansible core feature just merged
  implement conditionally only with BOTH group_by options

Make compat mode default be true
  in API models, UI add and edit controllers

Add several additional hostvars to translation
Add Azure tags null case translation

Make Azure group_by key off source_vars
  to be consistent with the script

support top-level ec2 boto_profile setting
2019-03-26 10:29:39 -04:00
AlanCoding
bc5881ad21 Primary development of inventory plugins, partial compat layer
Initialize some inventory plugin test data files
Implement openstack inventory plugin

This may be removed later:
- port non-JSON line strip method from core

Dupliate effort with AWX mainline devel
- Produce ansible_version related to venv

Refactor some of injector management, moving more
  of this overhead into tasks.py, when it comes to
  managing injector kwargs

Upgrade and move openstack inventory script
  sync up parameters

Add extremely detailed logic to inventory file creation
for ec2, Azure, and gce so that they are closer to a
genuine superset of what the contrib script used to give.
2019-03-26 10:29:39 -04:00
Jim Ladd
7cce3cad06 Add support for ec2 inventory plugin 2019-03-26 10:29:38 -04:00
AlanCoding
622fbc116b move script injection logic to inventory file 2019-03-26 10:29:38 -04:00
AlanCoding
b9d489c788 Use randomized file names for injector credential files 2019-03-26 10:29:38 -04:00
AlanCoding
5cbcfbe0c6 Port inventory source injector tests to functional tests
This new batch of tests assures that the injector logic
for inventory source in their old script version remains
untouched with the refactoring underway.

Plugins are also tested by the same means of comparing
to reference files, these will be used to assure that
all parameters that used to be respected are still
respected in the plugin system.
2019-03-26 10:29:38 -04:00
AlanCoding
43eff55fd4 fix bugs related to python3 2019-03-26 10:29:37 -04:00
AlanCoding
6c130fa6c3 Build-in inventory plugin code structure with gce working
supporting and related changes
 - Fix inconsistency between can_update / can_start
 - Avoid creating inventory file twice unnecessarily
 - Non-functional consolidation in Azure injection logic
 - Inject GCE creds as indented JSON for readability
 - Create new injector class structure, add gce
 - Reduce management command overrides of runtime environment
2019-03-26 10:29:35 -04:00
Yanis Guenane
fd50feb258 bwrap/runner: Add /etc/ssh in bind mounted folder
/etc/ssh is currently not bound when run into bwrap, this leads to
error like "Bad owner or permissions on /etc/ssh/ssh_config.d/05-redhat.conf"
since it cannot access this file.

https://github.com/ansible/awx/pull/3391 was done pre runner
integration.

Fixes: https://github.com/ansible/awx/issues/3392

Signed-off-by: Yanis Guenane <yanis@guenane.org>
2019-03-26 12:43:59 +01:00
Ryan Petrello
495dc2202f more iso cleanup and bug fixes 2019-03-25 17:47:58 -04:00
Ryan Petrello
af8e071840 remove old callback plugin code and tests 2019-03-25 12:26:51 -04:00
chris meyers
87e1ba4dea do not generate a random ident
* instead, set the ident passed to ansible runner to be the job id. That
way, on we know what directory to look in for results when the directory
structure is created.
2019-03-22 12:19:42 -04:00
chris meyers
b006510035 do not save sensitive env vars
* job_env gets exposed via the api. Sensitive env variables should be
redacted before saved into job_env.
2019-03-20 14:00:22 -04:00
chris meyers
1a6ae6e107 allow for runner setting parent_uuid
* Previously, parent_uuid was expected only on events generated for a
Job run. Now, there maybe a parent_uuid for any job type. AWX does not
support parenting events for any job type other than Job.
2019-03-20 11:05:01 -04:00
chris meyers
060585434a update tests 2019-03-20 09:44:38 -04:00
chris meyers
f7842cf283 refactor and fix unit tests
* fixup task TestGenericRun
* make runner callback functions accessable to testing
* reduce isintance() usage in run() by using build_ pattern
* move process_isolation param building to build_ function so it can be
tested
2019-03-18 14:21:47 -04:00
chris meyers
827ad0fa75 remove safe_args and add status_handler
* safe_args no longer makes sense. We have moved extra_vars to a file
and thus do not pass sensitive content on the cmdline
2019-03-18 14:21:47 -04:00