Commit Graph

16 Commits

Author SHA1 Message Date
Seth Foster
0ae9fe3624 if dependency fails, fail job in task manager 2022-05-12 14:00:13 -04:00
Alan Rominger
29d60844a8 Fix notification timing issue by sending in the latter of 2 events (#12110)
* Track host_status_counts and use that to process notifications

* Remove now unused setting

* Back out changes to callback class not needed after all

* Skirt the need for duck typing by leaning on the cached field

* Delete tests for deleted task

* Revert "Back out changes to callback class not needed after all"

This reverts commit 3b8ae350d218991d42bffd65ce4baac6f41926b2.

* Directly hardcode stats_event_type for callback class

* Fire notifications if stats event was never sent

* Remove test content for deleted methods

* Add placeholder for when no hosts matched

* Make field default be None, denote events processed with empty dict

* Make UI process null value for host_status_counts

* Fix tracking of EOF dispatch for system jobs

* Reorganize EVENT_MAP into class properties

* Consolidate conditional I missed from EVENT_MAP refactor

* Give up on the null condition, also applies for empty hosts

* Remove cls position argument not being used

* Move wrapup method out of class, add tests
2022-04-29 13:54:31 -04:00
Shane McDonald
ef0f6ca248 Merge pull request #11955 from shanemcd/fail-better
Increase resiliency when application crashes
2022-03-30 08:58:26 -04:00
Shane McDonald
b04747676c Provision the instance before booting anything
There was a race condition because the callback reciever tried to run this code:

  File "/awx_devel/awx/main/management/commands/run_callback_receiver.py", line 31, in handle
    CallbackBrokerWorker(),
  File "/awx_devel/awx/main/dispatch/worker/callback.py", line 49, in __init__
    self.subsystem_metrics = s_metrics.Metrics(auto_pipe_execute=False)
  File "/awx_devel/awx/main/analytics/subsystem_metrics.py", line 156, in __init__
    self.instance_name = Instance.objects.me().hostname

Before get_or_register was being called by the dispatcher.
2022-03-29 14:07:49 -04:00
Jeff Bradberry
6c1adade25 Merge pull request #11947 from jbradberry/django-3.2-upgrade
Remove the out-of-band JSONField migration
2022-03-28 12:02:53 -04:00
Jeff Bradberry
d54838cd94 Remove the out-of-band migration
that was turning all old JSONFields into a jsonb type database column.
The use of JSONBlob makes this unnecessary.
2022-03-24 15:21:59 -04:00
Chris Meyers
59bd73bff8 add setting for notification job status retry loop
* We trigger notifications when the callback receiver processes the
playbook_on_stats event. This is the last event in ansible-playbook and
the process should exist very shortly after this event is emitted. The
trouble comes in with the isolated node feature. There is a management
playbook that runs periodically that pulls the events from the remote
node. It's possible that the management playbooks runs, gets the
playbook_on_stats event, but does not see that the playbook is finished
running. Therefore the job status is still seen as 'running' BUT we have
kicked of the notification for the job. The notification worker will
enter a loop waiting on the job to enter the finished state. In this
case the time it takes for the job to enter the finished state can be
long, roughly 2 * the management playbook run time.
* This new setting allows the user to increase the time that the
notification spends waiting for the job to enter the finished state.
2022-03-22 09:20:14 -04:00
Jeff Bradberry
ac6a82eee4 Merge pull request #11654 from jbradberry/django-3.2-upgrade
Django 3.2 upgrade
2022-03-17 10:34:22 -04:00
Alan Rominger
2321f06c8a Only clear hop node errors if they were lost before, not lost now 2022-03-14 16:04:55 -04:00
Alan Rominger
99bbc347ec Fill in errors for hop nodes when Last Seen is out of date, and clear them when not (#11714)
* Process unresponsive and newly responsive hop nodes

* Use more natural way to zero hop node capacity, add test

* Use warning as opposed to warn for log messages
2022-03-09 13:21:32 -05:00
Jeff Bradberry
676b8f6d8f Implement an out-of-band migration to change the json fields 2022-03-07 18:11:36 -05:00
Jeff Bradberry
b852baaa39 Fix up logger .warn() calls to use .warning() instead
This is a usage that was deprecated in Python 3.0.
2022-03-07 18:11:36 -05:00
Jeff Bradberry
a3a216f91f Fix up new Django 3.0 deprecations
Mostly text based: force/smart_text, ugettext_*
2022-03-07 18:11:36 -05:00
Jeff Bradberry
7c550a76a5 Make sure to filter out control-plane nodes in inspect_execution_nodes
Also, make sure that the cluster host doesn't get marked as lost by
this machinery.
2022-01-25 11:06:20 -05:00
Jeff Bradberry
334c33ca07 Handle receptorctl advertisements for hop nodes
counting it towards their heartbeat.  Also, leave off the link to the
health check endpoint from hop node Instances.
2022-01-24 16:51:45 -05:00
Amol Gautam
a4a3ba65d7 Refactored tasks.py to a package
--- Added 3 new sub-package : awx.main.tasks.system , awx.main.tasks.jobs , awx.main.tasks.receptor
--- Modified the functional tests and unit tests accordingly
2022-01-14 11:55:41 -05:00