Commit Graph

1039 Commits

Author SHA1 Message Date
Ryan Petrello
013b411a0a remove old fact scanning plugins 2020-09-04 10:51:34 -04:00
Ryan Petrello
3edaa6bc14 remove some dead code 2020-09-01 15:26:38 -04:00
Alan Rominger
99aff93930 Get rid of ansible version checking 2020-09-01 12:50:59 -04:00
Chris Meyers
a9cdf07690 push global invsource fields onto invsource obj 2020-09-01 12:50:58 -04:00
Alan Rominger
0b371b4340 Copy library folder to job private data dir
Remove inventory scripts show because they no longer exist

Remove reference to non-existent callback directory

Remove more references to removed path
2020-08-31 22:14:59 -04:00
Alan Rominger
e811711a49 Put project playbook in runner project folder 2020-08-27 11:07:10 -04:00
Philip Douglass
6dc41f54fc Ensure scm credentials are passed for archive scm type 2020-08-22 11:16:04 -04:00
AlanCoding
ddb8c93f17 Fix log message 2020-07-21 11:21:21 -04:00
AlanCoding
425c70826f code style edit, always use cache_id as property in tasks 2020-07-21 10:50:13 -04:00
AlanCoding
8195bd7386 Remove job pre-creation of content folders 2020-07-20 19:45:22 -04:00
AlanCoding
300d54864d Remove project update vars no longer used 2020-07-20 19:45:21 -04:00
AlanCoding
fefab89815 Integrate content caching with existing task logic
Revert the --force flags

use the update id as metric for role caching

Shift the movement of cache to job folder from rsync task to python

Only install roles and collections if needed

Deal with roles and collections for jobs without sync
Skip local copy if roles or collections turned off

update docs for content caching

Design pivot - use empty cache dir to indicate lack of content

Do not cache content if we did not install content

Test changes to allay concerns about reliability of local_path

Do not blow away cache for SCM inventory updates
2020-07-20 19:45:21 -04:00
Bill Nottingham
1f4c46d873 Invalidate the cache on a new project revision 2020-07-20 19:45:18 -04:00
Bill Nottingham
d272ee3521 Cache downloaded roles & collections
Populate the cache the first time the job is run for a revision
that needs them, and for future runs for that revision just
copy it into the private directory.

Delete the cache on project deletion.
2020-07-20 19:45:15 -04:00
Ryan Petrello
78229f5871 replace certain terms with more inclusive language
see: https://www.redhat.com/en/blog/making-open-source-more-inclusive-eradicating-problematic-language
2020-06-30 10:49:19 -04:00
AlanCoding
3c8c1858d4 Fix migration through manual testing 2020-06-17 14:25:15 -04:00
AlanCoding
71cf3198d8 Add data migration to convert cloudforms scripts to SCM 2020-06-17 14:25:15 -04:00
AlanCoding
b0ff763a0b flake8 fixes 2020-06-17 14:25:15 -04:00
AlanCoding
bedb1f364d Remove inventory scripts 2020-06-17 14:25:11 -04:00
Jim Ladd
3b61b82616 wrap --instance-id-var in quotes 2020-06-10 14:06:57 -07:00
Ryan Petrello
64ebfa75a8 remove a highly verbose log lines 2020-06-10 11:50:13 -04:00
Bill Nottingham
17139f1f82 Always check configuration before gathering data.
We shouldn't perform expensive operations if we won't be able to send it.
Only log at debug level, otherwise every node will log this every 5
minutes.
2020-05-20 11:03:13 -04:00
Christian Adams
19ccb5e213 Mark job_explanation strings after they are read from the db
- For strings that need to be translated, but are saved in the db:
   * They must be marked for translation using gettext_noop() to be
   translated.
   * And must also be marked for translation with _() when read from db
   and shown to the user.
   * [Ref]: https://docs.djangoproject.com/en/3.0/topics/i18n/translation/#marking-strings-as-no-op
2020-05-15 22:50:50 -04:00
Seth Foster
4da0e0dd80 Vendor collections for isolated jobs to work in ansible 2.10
kubectl and synchronize are now part of community.kubernetes
and ansible.posix collections, respectively. This change installs
these collections to a local directory to be used in inventory and
isolated management playbooks.

awx issue #6930
2020-05-13 10:41:01 -04:00
Ryan Petrello
e7347d15c1 drastically optimize job host summary creation
see: https://github.com/ansible/awx/issues/6991
2020-05-11 17:13:06 -04:00
softwarefactory-project-zuul[bot]
1ca3d1fe3b Merge pull request #6979 from wenottingham/nobody-pexpects-a-timeout-inquisition
Remove pexpect_timeout setting for ansible-runner

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-05-11 15:33:07 +00:00
Ryan Petrello
086d6951b4 Merge branch 'downstream' into devel 2020-05-08 16:48:48 -04:00
Bill Nottingham
8cdffd0dd0 Remove pexpect_timeout setting for ansible-runner
PEXPECT_TIMEOUT has never(?) been an actual setting, and 5 is the
runner default anyway.
2020-05-08 16:31:11 -04:00
Bill Nottingham
4c6cac90fd Remove ssh version checking.
6.6 was added in RHEL 7.1, which is well before our supported platforms.
2020-05-07 15:30:55 -04:00
Christian Adams
e41d33991a Prevent a race condition when writing the rsyslog.conf 2020-05-05 15:55:08 -04:00
Ryan Petrello
2997911fd4 Merge branch 'downstream' into devel 2020-05-01 13:53:30 -04:00
Ryan Petrello
bf65b40241 only sanitize project update events for the scm modules
these are the only modules in the project update playbook that actually
utilize the SCM URL (which is what potentially contains sensitive data)
2020-05-01 11:39:46 -04:00
Ryan Petrello
aad371d224 Merge branch 'downstream' into devel 2020-04-28 12:18:13 -04:00
Alan Rominger
41014e62b7 Avoid applying galaxy settings if no user settings exist (#4262) 2020-04-27 16:15:11 -04:00
softwarefactory-project-zuul[bot]
88f44cb185 Merge pull request #6811 from wenottingham/ssh_aaaaaaarghs
Don't override ANSIBLE_SSH_ARGS for ad-hoc commands.

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-27 16:44:23 +00:00
Bill Nottingham
3f5820adf8 Don't override ANSIBLE_SSH_ARGS for ad-hoc commands.
This reverts 8beed7f4, which says:
    Disable ControlPersist for ad hoc commands, should avoid any
    issues with proot or needing to clean up sockets afterwards.

Given we've switched to the much less finicky bwrap for process
isolation, along with runner-based process killing, this probably
isn't needed any more.
2020-04-22 20:46:18 -04:00
AlanCoding
fcf75af6a7 Get current cloud sources working from collection
update test data files

Adopt official vendor location

openstack not published yet

Add collections to show paths

Add collections loc to installer settings

Add vendored collections to show path again
2020-04-16 20:55:59 -04:00
Ryan Petrello
bba680671b when writing the rsyslog config, do it post-commit
there's a race condition if we do this pre-commit where the correct
value isn't actually *persisted* to the database yet, and we end up
saving the *prior* setting values
2020-04-13 11:44:00 -04:00
Christian Adams
d31c528257 Fix Logging settings "Test" button functionality 2020-04-13 11:43:59 -04:00
Shane McDonald
c0af3c537b Configure rsyslog to listen over a unix domain socket instead of a port
- Add a placeholder rsyslog.conf so it doesn't fail on start
 - Create access restricted directory for unix socket to be created in
 - Create RSyslogHandler to exit early when logging socket doesn't exist
 - Write updated logging settings when dispatcher comes up and restart rsyslog so they  take effect
 - Move rsyslogd to the web container and create rpc supervisor.sock
 - Add env var for supervisor.conf path
2020-04-13 11:43:59 -04:00
Ryan Petrello
589d27c88c POC: replace our external log aggregation feature with rsyslog
- this change adds rsyslog (https://github.com/rsyslog/rsyslog) as
  a new service that runs on every AWX node (managed by supervisord)
  in particular, this feature requires a recent version (v8.38+) of
  rsyslog that supports the omhttp module
  (https://github.com/rsyslog/rsyslog-doc/pull/750)
- the "external_logger" handler in AWX is now a SysLogHandler that ships
  logs to the local UDP port where rsyslog is configured to listen (by
  default, 51414)
- every time a LOG_AGGREGATOR_* setting is changed, every AWX node
  reconfigures and restarts its local instance of rsyslog so that its
  fowarding settings match what has been configured in AWX
- unlike the prior implementation, if the external logging aggregator
  (splunk/logstash) goes temporarily offline, rsyslog will retain the
  messages and ship them when the log aggregator is back online
- 4xx or 5xx level errors are recorded at /var/log/tower/external.err
2020-04-13 11:43:59 -04:00
Ryan Petrello
6b4219badb more ansible runner isolated cleanup
follow-up to https://github.com/ansible/awx/pull/6296
2020-04-08 01:18:05 -04:00
Ryan Petrello
301d6ff616 make the job event bigint migration chunk size configurable 2020-03-27 09:28:10 -04:00
Ryan Petrello
c8044b4755 migrate event table primary keys from integer to bigint
see: https://github.com/ansible/awx/issues/6010
2020-03-26 15:54:38 -04:00
Bill Nottingham
ac68e8c4fe Preserve symlinks when copying a tree.
This avoids creating a recursive symlink tree.
2020-03-20 13:41:16 -04:00
chris meyers
47f5c17b56 log when notifications fail to send
* If a job does not finish in the 5 second timeout. Let the user know
that we failed to even try to send the notification.
2020-03-20 09:11:01 -04:00
Ryan Petrello
d40a5dec8f change when we send job notifications to avoid a race condition
success/failure notifications for *playbooks* include summary data about
the hosts in based on the contents of the playbook_on_stats event

the current implementation suffers from a number of race conditions that
sometimes can cause that data to be missing or incomplete; this change
makes it so that for *playbooks* we build (and send) the notification in
response to the playbook_on_stats event, not the EOF event
2020-03-19 10:01:52 -04:00
Ryan Petrello
1caa2e0287 work around a limitation in postgres notify to properly support copying
postgres has a limitation on its notify message size (8k), and the
messages we generate for deep copying functionality easily go over this
limit; instead of passing a giant nested data structure across the
message bus, this change makes it so that we temporarily store the JSON
structure in memcached, and look it up from *within* the task

see: https://github.com/ansible/tower/issues/4162
2020-03-18 16:10:20 -04:00
chris meyers
dc6c353ecd remove support for multi-reader dispatch queue
* Under the new postgres backed notify/listen message queue, this never
actually worked. Without using the database to store state, we can not
provide a at-most-once delivery mechanism w/ multi-readers.
* With this change, work is done ONLY on the node that requested for the
work to be done. Under rabbitmq, the node that was first to get the
message off the queue would do the work; presumably the least busy node.
2020-03-18 16:10:16 -04:00
chris meyers
2a2c34f567 combine all the broker replacement pieces
* local redis for event processing
* postgres for message broker
* redis for websockets
2020-03-18 16:10:15 -04:00