Commit Graph

210 Commits

Author SHA1 Message Date
chris meyers
bd7d9db1ce correctly cascade set null
* It's problematic to delete an instance that is referenced by a foreign
key; where the referening model is one that has a Polymorphic parent.
* Specifically, when Django goes to nullify the relationship it relies
on the related instances[0] class type to issue a query to decide what
to nullify. So if the foreignkey references multiple different types
(i.e. ProjectUpdate, Job) then only 1 of those class types will get
nullified. The end result is an IntegrityError when delete() is called.
* This changeset ensures that the parent Polymorphic class is queried so
that all the foreignkey entries are nullified
* Also remove old Django "hack" that doesn't work with Django 1.11
2018-04-06 11:10:16 -04:00
AlanCoding
98dc59765e exclude last_used from activity stream 2018-03-28 12:53:11 -04:00
AlanCoding
8c167e50c9 Continuously stream data from verbose jobs
In verbose unified job models (inventory updates, system jobs,
etc.), do not delay dispatch just because the encoded
event data is not part of the data written to the buffer.

This allows output from these commands to be submitted
to the callback queue as they are produced, instead
of waiting until the buffer is closed.
2018-03-28 11:05:49 -04:00
Chris Meyers
3446134501 Merge pull request #1646 from chrismeyersfsu/fix-kombu_unicode
use non-unicode queue names
2018-03-21 21:59:05 -04:00
chris meyers
e0803b9f08 use non-unicode queue names
* Use unicode InstanceGroup and queue names up until the point we
actually create the queue
* kombu add_consumers returns a dict with a value that contians the
passed in queue name. Trouble is, the returned dict value is a string
and not a unicode string and this results in an error.
2018-03-21 16:50:07 -04:00
Ryan Petrello
6a96e6a268 dynamically set worker autoscale max_concurrency based on system memory 2018-03-21 11:10:48 -04:00
Chris Meyers
2640ef8b1c Merge pull request #1536 from chrismeyersfsu/fix-protect_instance_groups
prevent instance group delete if running jobs
2018-03-15 14:57:45 -04:00
chris meyers
5d5d8152c5 prevent instance group delete if running jobs
* related to https://github.com/ansible/ansible-tower/issues/7936
2018-03-15 14:25:49 -04:00
Matthew Jones
b0cf4de072 Implement container-cluster aware capacity determination
* Added two settings values for declaring absolute cpu and memory
  capacity that will be picked up by the capacity utility methods
* installer inventory variables for controlling the amount of cpu and
  memory container requests/limits for the awx task containers
* Added fixed values for cpu and memory container requests for other
  containers
* configmap uses the declared inventory variables to define the
  capacity inputs that will be used by AWX to correspond to the same
  inputs for requests/limits on the deployment.
2018-03-14 14:35:45 -04:00
Matthew Jones
acde2520d0 Sort cloud regions in a stable way
* All comes first
* Then US regions
* Then all other regions alphabetically
2018-03-13 15:31:28 -04:00
Alan Rominger
dcae4f65b5 Merge pull request #1330 from AlanCoding/capable_of_anything
New copy fields, clean up user_capabilities logic
2018-03-13 12:05:45 -04:00
chris meyers
e2ed1542e6 more celery rollback
* Setting reload code calls a celery 4.x method signature. This changes
it back to a 3.x safe call.
2018-03-09 09:27:09 -05:00
adamscmRH
701a5c9a36 hides client_secret from act stream 2018-03-02 14:47:49 -05:00
Chris Meyers
d551566b4d Merge pull request #1372 from chrismeyersfsu/old-celery3
celery 4.x to 3.x roll back
2018-02-27 15:26:46 -05:00
chris meyers
6606a29f57 celery 4.x -> 3.x change route config name 2018-02-27 14:13:05 -05:00
AlanCoding
ce9234df0f Revamp user_capabilities with new copy fields
Add copy fields corresponding to new server-side copying

Refactor the way user_capabilities are delivered
 - move the prefetch definition from views to serializer
 - store temporary mapping in serializer context
 - use serializer backlinks to denote polymorphic prefetch model exclusions
2018-02-26 12:13:41 -05:00
Matthew Jones
8505783350 Merge remote-tracking branch 'tower/release_3.2.3' into devel
* tower/release_3.2.3:
  fix unicode bugs with log statements
  use --export option for ansible-inventory
  add support for new "BECOME" prompt in Ansible 2.5+ for adhoc commands
  enforce strings for secret password inputs on Credentials
  fix a bug for "users should be able to change type of unused credential"
  fix xss vulnerabilities - on host recent jobs popover - on schedule name tooltip
  fix a bug when testing UDP-based logging configuration
  bump templates form credential_types page limit
  Wait for Slack RTM API websocket connection to be established
  don't process artifacts from custom `set_stat` calls asynchronously
  don't overwrite env['ANSIBLE_LIBRARY'] when fact caching is enabled
  only allow facts to cache in the proper file system location
  replace our memcached-based fact cache implementation with local files
  add support for new "BECOME" prompt in Ansible 2.5+
  fix a bug in inventory generation for isolated nodes
  properly handle unicode for isolated job buffers
2018-02-20 12:22:25 -05:00
Seth Jennings
42ff1cfd67 add import_playbook as top-level playbook indicator 2018-02-19 16:03:08 -06:00
Matthew Jones
0d2daecf49 Merge pull request #1243 from matburt/fix_clustering_isolated
Fix isolated instance clustering implementation
2018-02-14 08:32:24 -05:00
Matthew Jones
ffe5a92eb9 Update isolated instance capacity calculaltion 2018-02-13 21:51:50 -05:00
Matthew Jones
925d9efecf Fixing up isolated node execution after cluster changes
* Rework queue detection to include control groups and isolated instances
* Fix up development tooling around isolated nodes
* Update unit tests
2018-02-13 21:51:38 -05:00
Chris Church
67ec811e8d Merge pull request #1186 from cclauss/execfile-file-reduce-StandardError
Miscellaneous Python 3 changes: execfile(), file(), reduce(), StandardError
2018-02-13 15:11:24 -05:00
Ryan Petrello
194c2dcf0b improve a bwrap test 2018-02-12 10:14:37 -05:00
Ryan Petrello
83b5377387 Merge pull request #1187 from ryanpetrello/file-your-vars-away-for-a-rainy-day
pass extra vars via file rather than via commandline
2018-02-12 08:48:19 -05:00
cclauss
2e623ad80c Change unicode() --> six.text_type() for Python 3 2018-02-11 21:09:12 +01:00
Bill Nottingham
aa5bd9f5bf Pass extra vars via file rather than via commandline, including custom creds.
The extra vars file created lives in the playbook private runtime
directory, and will be reaped along with the rest of the directory.

Adjust assorted unit tests as necessary.
2018-02-10 09:27:24 -05:00
cclauss
260aec543e Misc Python 3 changes: execfile(), file(), reduce(), StandardError 2018-02-09 17:17:05 +01:00
cclauss
c371b869dc basestring to six.string_types for Python 3 2018-02-09 16:28:36 +01:00
Bill Nottingham
c1a0e2cd16 Have bubblewrap mount a new /proc in the wrapped environment.
Since we're running with a new pid namespace, we should have
a new /proc that is in that namespace. Otherwise things will
be weird.
2018-02-07 15:47:03 -05:00
Matthew Jones
70bf78e29f Apply capacity algorithm changes
* This also adds fields to the instance view for tracking cpu and
  memory usage as well as information on what the capacity ranges are
* Also adds a flag for enabling/disabling instances which removes them
  from all queues and has them stop processing new work
* The capacity is now based almost exclusively on some value relative
  to forks
* capacity_adjustment allows you to commit an instance to a certain
  amount of forks, cpu focused or memory focused
* Each job run adds a single fork overhead (that's the reasoning
  behind the +1)
2018-02-01 16:57:09 -05:00
Matthew Jones
d9e774c4b6 Updates for automatic triggering of policies
* Switch policy router queue to not be "tower" so that we don't
  fall into a chicken/egg scenario
* Show fixed policy list in serializer so a user can determine if
  an instance is manually managed
* Change IG membership mixin to not directly handle applying topology
  changes. Instead it just makes sure the policy instance list is
  accurate
* Add create/delete hooks for instances and groups to trigger policy
  re-evaluation
* Update policy algorithm for fairer distribution
* Fix an issue where CELERY_ROUTES wasn't renamed after celery/django
  upgrade
* Update unit tests to be more explicit
* Update count calculations used by algorithm to only consider
  non-manual instances
* Adding unit tests and fixture
* Don't propagate logging messages from awx.main.tasks and
  awx.main.scheduler
* Use advisory lock to prevent policy eval conflicts
* Allow updating instance groups from view
2018-02-01 16:56:16 -05:00
Chris Meyers
c9ff3e99b8 celeryd attach to queues dynamically
* Based on the tower topology (Instance and InstanceGroup
relationships), have celery dyamically listen to queues on boot
* Add celery task capable of "refreshing" what queues each celeryd
worker listens to. This will be used to support changes in the topology.
* Cleaned up some celery task definitions.
* Converged wrongly targeted job launch/finish messages to 'tower'
queue, rather than a 1-off queue.
* Dynamically route celery tasks destined for the local node
* separate beat process

add support for separate beat process
2018-02-01 16:37:33 -05:00
Ryan Petrello
982539f444 fix a bug when testing UDP-based logging configuration
see: https://github.com/ansible/ansible-tower/issues/7868
2018-01-29 12:05:51 -05:00
Wayne Witzel III
55a616cba6 Load Celery inspector manually when needed 2018-01-29 14:57:03 +00:00
Ryan Petrello
5387846cbb Merge pull request #992 from ryanpetrello/optimize-output-event-filter
optimize OutputEventFilter for large stdout streams
2018-01-17 14:24:15 -05:00
Chris Meyers
e33265e12c add job_id to fact cache log output 2018-01-17 10:19:27 -05:00
Ryan Petrello
51f7907a01 optimize OutputEventFilter for large stdout streams
update our event data search algorithm to be a bit lazier in event data
discovery; this drastically improves processing speeds for stdout >5MB

see: https://github.com/ansible/awx/issues/417
2018-01-16 14:41:35 -05:00
Ryan Petrello
1e8c89f536 implement support for per-playbook/project/org virtualenvs
see: https://github.com/ansible/awx/issues/34
2018-01-09 22:47:01 -05:00
Ryan Petrello
c6b6a3ad89 send and subcribe to summary events for all new events 2018-01-03 09:09:45 -05:00
Ryan Petrello
0b30e7907b change stdout composition to generate from job events on the fly
this approach totally removes the process of reading and writing stdout
files on the local file system at settings.JOBOUTPUT_ROOT when jobs are
run; now stdout content is only written on-demand as it's fetched for
the deprecated `stdout` endpoint

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:43 -05:00
AlanCoding
c8e10adc96 fix bug saving extra_data and follow prompts rules
display_extra_vars was not taking a copy of the data before
acting on it - this causes a bug where the activity stream will
modify the existing object on the model. That leads to new data
not being accepted.

Also moved the processing of extra_data to prior to the accept
or ignore kwargs logic so that we pass the right (post-encryption)
form of the variables.
2017-12-18 10:50:22 -05:00
AlanCoding
74bf058d62 encrypt password answers on config save 2017-12-15 07:48:55 -05:00
AlanCoding
8a2a5b0fb1 avoid slowdown generating smart_filter 2017-12-14 09:39:39 -05:00
Matthew Jones
9dbcc5934e Merge remote-tracking branch 'tower/release_3.2.2' into devel 2017-12-13 12:25:47 -05:00
AlanCoding
72a8854c27 Make ask_mapping a simple class property
from PR feedback of saved launchtime configurations
2017-12-08 13:45:23 -05:00
AlanCoding
34a8e0a9b6 Feature: saved launchtime configurations
Consolidate prompts accept/reject logic in unified models
Break out accept/reject logic for variables
Surface new promptable fields on WFJT nodes, schedules

Make schedules and workflows accurately reject variables
  that are not allowed by the prompting
  rules or the survey rules on the template

Validate against unallowed extra_data in system job schedules
Prevent schedule or WFJT node POST/PATCH with unprompted data
Move system job days validation to new mechanism
Add new psuedo-field for WFJT node credential
Add validation for node related credentials
Add related config model to unified job
Use JobLaunchConfig model for launch RBAC check

Support credential overwrite behavior with multi-creds
  change modern manual launch to use merge behavior
Refactor JobLaunchSerializer, self.instance=None
Modularize job launch view to create "modern" data
Auto-create config object with every job
Add create schedule endpoint for jobs
2017-12-08 13:38:54 -05:00
AlanCoding
67867cf0c8 flake8: comply with new E722 rule 2017-12-01 12:16:44 -05:00
Wayne Witzel III
96904968d8 Fix migration issues, tests, and templates 2017-11-09 17:29:48 -05:00
AlanCoding
b4a446dba0 raise error for invalid type lookup 2017-10-26 11:25:40 -04:00
AlanCoding
90f63774f4 flake8: comply with new E722 rule 2017-10-23 14:36:48 -04:00