Commit Graph

472 Commits

Author SHA1 Message Date
Ryan Petrello
301d6ff616 make the job event bigint migration chunk size configurable 2020-03-27 09:28:10 -04:00
chris meyers
770b457430 redis socket support 2020-03-18 16:10:19 -04:00
chris meyers
d6594ab602 add broadcast websocket metrics
* Gather brroadcast websocket metrics and push them into redis every
configurable seconds.
* Pop metrics from redis in web view layer to display via the api on
demand
2020-03-18 16:10:18 -04:00
chris meyers
b6b9802f9e increase per-channel capacity
* 100 is the default capacity for a channel. If the client doesn't read
the socket fast enough, websocket messages can and will be lost. This
increases the default to 10,000
2020-03-18 16:10:18 -04:00
chris meyers
3b9e67ed1b remove channel group model
* Websocket user session <-> group subscription membership now resides
in Redis rather than the database.
2020-03-18 16:10:18 -04:00
chris meyers
3c5c9c6fde move broadcast websocket out into its own process 2020-03-18 16:10:18 -04:00
chris meyers
03b73027e8 websockets aware of Instance changes
* New tower nodes that are (de)registered in the Instance table are seen
by the websocket layer and connected to or disconnected from by the
websocket broadcast backplane using a polling mechanism.
* This is especially useful for openshift and kubernetes. This will be
useful for standalone Tower in the future when the restarting of Tower
services is not required.
2020-03-18 16:10:17 -04:00
chris meyers
be58906aed remove kombu 2020-03-18 16:10:17 -04:00
chris meyers
dc6c353ecd remove support for multi-reader dispatch queue
* Under the new postgres backed notify/listen message queue, this never
actually worked. Without using the database to store state, we can not
provide a at-most-once delivery mechanism w/ multi-readers.
* With this change, work is done ONLY on the node that requested for the
work to be done. Under rabbitmq, the node that was first to get the
message off the queue would do the work; presumably the least busy node.
2020-03-18 16:10:16 -04:00
chris meyers
2a2c34f567 combine all the broker replacement pieces
* local redis for event processing
* postgres for message broker
* redis for websockets
2020-03-18 16:10:15 -04:00
chris meyers
c8eeacacca POC channels 2 2020-03-18 16:10:12 -04:00
Ryan Petrello
38a08d163c get rid of celery/celerybeat
alternative to https://github.com/ansible/awx/pull/2530 which makes use
of https://pypi.org/project/schedule/

this doesn't have support for any persistence (like how celery beat uses
a shelve file), because all of our periodic jobs run at most every few
minutes
2020-02-10 17:32:02 -05:00
Ryan Petrello
78b00652bd add the ability to enable profiling for the callback receiver workers 2020-01-27 12:03:53 -05:00
softwarefactory-project-zuul[bot]
23b2b136d6 Merge pull request #5707 from AlanCoding/bulk_create_logs
Allow CTiT log level to log bulk_create lines

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-01-22 15:04:17 +00:00
Bill Nottingham
44e176dde8 Change how analytics is gathered to only gather once per interval 2020-01-21 11:40:51 -05:00
Ben Thomasson
652a428438 Fixes crontab for gather_analytics to run once every 4 hours 2020-01-20 13:30:10 -05:00
AlanCoding
ceed6f8d9b Allow CTiT log level to log bulk_create lines 2020-01-20 12:41:10 -05:00
Ryan Petrello
306f504fb7 optimize the callback receiver to buffer writes on high throughput
additionaly, optimize away several per-event host lookups and
changed/failed propagation lookups

we've always performed these (fairly expensive) queries *on every event
save* - if you're processing tens of thousands of events in short
bursts, this is way too slow

this commit also introduces a new command for profiling the insertion
rate of events, `awx-manage callback_stats`

see: https://github.com/ansible/awx/issues/5514
2020-01-14 12:04:26 -05:00
Seth Foster
7873d08311 Update pip and setuptools in requirements txt
Versions selected to be pre-19 pip
due to unresolved issues with the build systems

Upgrade everything, party on

document new process

rotate license files

fix Swagger schema generation target

Remove --ignore-installed flag
2020-01-07 17:14:32 -06:00
Ryan Petrello
4a6147d4c2 add the ability to generate dot graphs for per-request profiling 2020-01-04 07:09:42 -05:00
Graham Mainwaring
055c02072f Default LOGIN_REDIRECT_URL should be blank, not null 2019-12-18 15:13:46 -05:00
Graham Mainwaring
7700050d10 Add default for LOGIN_REDIRECT_OVERRIDE 2019-12-11 17:21:02 -05:00
Alan Rominger
8e7d607a47 Only turn off Galaxy cert verification via toggle (#3933) 2019-11-19 08:54:40 -05:00
Bill Nottingham
5cdf2f88da Remove admin alerts, there are better mechanisms for this 2019-10-30 19:35:45 -04:00
Graham Mainwaring
b2b33605cc Add UI toggle to disable public Galaxy (#3867) 2019-10-29 11:24:11 -04:00
Ryan Petrello
16812542f8 implement a simple periodic pod reaper for container groups
see: https://github.com/ansible/awx/issues/4911
2019-10-17 17:06:36 -04:00
Shane McDonald
8f75382b81 Implement retry logic for container group pod launches 2019-10-10 15:53:56 -04:00
Christian Adams
844b8a803f update frequency of collection for automation analytics 2019-10-09 21:10:48 -04:00
Ryan Petrello
a076e84a33 add a settings flag for writing all external logs to disk 2019-10-09 13:54:54 -04:00
AlanCoding
c09039e963 Add setting for auth_url
Also adjust public galaxy URL setting to
allow using only the primary Galaxy server

Include auth_url in token exclusivity validation
2019-10-07 14:02:43 -04:00
AlanCoding
922e779a86 Rename private to primary in galaxy settings
use a setting for the public galaxy URL
2019-10-07 14:02:43 -04:00
AlanCoding
093bf6877b Finish adding settings to UI 2019-10-07 14:02:42 -04:00
AlanCoding
d59d8562db Avoid redacting Galaxy URLs 2019-10-07 14:02:42 -04:00
AlanCoding
c566c332f9 Initial env var implementation of private galaxy server 2019-10-07 14:02:42 -04:00
Ryan Petrello
b45b9333e1 Merge pull request #4716 from jladdjr/perf_stats
Enable collection of performance stats
2019-10-04 17:09:30 -04:00
Jim Ladd
cf1c9a0559 Add awx settings for resource profiling 2019-10-04 12:48:29 -07:00
Shane McDonald
bd5003ca98 Task manager / scheduler Kubernetes integration 2019-10-04 13:21:21 -04:00
Seth Foster
db1ad2de95 Set REFRESH_TOKEN_EXPIRE_SECONDS
- Set OAUTH2 REFRESH_TOKEN_EXPIRE_SECONDS to 1 month
  (2628000 seconds)
- If not set, awx-manage cleartokens, or cleanup_tokens,
  will not work properly
- Once cleartokens is run, this setting is the amount of
  time after an access token expires that we keep its
  refresh token in the database
2019-10-02 13:51:46 -04:00
Ryan Petrello
27c4e35ee4 warn loudly if celerybeat encounters AMQP connection issues
related: https://github.com/ansible/awx/pull/4857
2019-10-01 09:32:22 -04:00
Ryan Petrello
b80eafe4a1 make kombu DNS failures louder in the logs 2019-09-30 16:48:09 -04:00
Ryan Petrello
82be87566f improve host key checking configurability
see: https://github.com/ansible/tower/issues/3737
2019-09-30 14:13:07 -04:00
Bill Nottingham
0fb7c859f4 Revert 9b95cc27c4.
We do not want to create a new setting for a on-Tower-host global
collection path at this time.
2019-09-04 11:43:33 -04:00
chris meyers
4be65a0879 collections/requirements.yml support
* just like we support ansible-galaxy role install, support
ansible-galaxy collection install
2019-08-12 22:14:30 -04:00
AlanCoding
63978f7d10 remove setting not actually customizable 2019-08-06 13:40:52 -04:00
chris meyers
9b95cc27c4 map in a global collections path 2019-07-17 15:36:09 -04:00
Jeff Bradberry
f94959d120 Remove the custom get_view_name function
It appeared to not be doing anything that we were making use of that
couldn't already be done, slightly differently, using DRF's built-in
one.
2019-06-12 14:12:59 -04:00
Ryan Petrello
40b1e89b67 add the ability to disable RabbitMQ queue durability 2019-05-28 15:49:32 -04:00
AlanCoding
d3f047d731 Move dynamic log level logic to filter 2019-05-22 14:00:39 -04:00
Jeff Bradberry
44907b33dc Fix the middleware ordering 2019-05-17 14:40:33 -04:00
Christian Adams
2b327935de reorder migrations to avoid session error 2019-05-16 15:13:47 -04:00