This document summarizes a ManageIQ sprint review covering Sprint 49 from November 14, 2016. It provides statistics on pull requests and backports. It also summarizes work done in areas like the classic UI, providers, automation, API, performance, and quality engineering. A new manageiq-performance gem was introduced to help with performance monitoring.

1. ManageIQ
Sprint 49 Review - Sprint End November 14, 2016
November 16, 2016

2. Overview
● Sprint Statistics (Oleg Barenboim)
● Community Update (Carol Chen)
● Classic UI (Dan Clarizio)
● Service UI (Chris Kacerguis)
● Providers (Adam Grare)
● Automate (Greg McCullough)
● Platform (Gregg Tanzillo)
● API (Alberto Bellotti)
● Performance (Dennis Metzger)
● Quality Engineering (Dave Johnson)
● Discussion

3. Sprint Statistics
(O. Barenboim)
354 Pull Requests Merged for ManageIQ/manageiq

4. Sprint Statistics - Euwe Backports
(O. Barenboim)
162 Pull Requests Backported to Euwe

5. PR Breakdown by Feature Category
(O. Barenboim)
* Note that some PRs
have more than one
category.
Providers

6. Sprint Comparison
(O. Barenboim)
Darga-2
DargaBeta
Darga-3
DargaGA
DargaRC
Darga-4
EuweBeta
EuweRC

7. All Repo Stats
(O. Barenboim) Repo Closed
manageiq 354
integration_tests 118
manageiq-ui-service 35
manageiq.org 14
manageiq-providers-amazon 12
azure-armest 9
manageiq-api-client 7
manageiq-pods 7
manageiq-providers-lenovo 7
manageiq-providers-azure 6
manageiq-appliance-build 5
wrapanapi 5
guides 4
miq_bot 3
manageiq-appliance 2
manageiq-gems-pending 2
ui-components 2
Total 592

8. Community Update
(Carol Chen)
● 82% more
pageviews than
previous 4-week
period
● 48% pageviews
are to /docs/
pages (compared
to 28% in previous
period)

9. Community Update
● Recent events
○ KubeCon (Nov 8-9, Seattle) - Jerome Marc with ManageIQ demo
○ RubyConf (Nov 10-12, Cincinnati) - Chris Arcand presented “Deletion Driven Development”
● Upcoming events
○ DevConf.CZ (Jan 27-29, 2017, Brno)
○ FOSDEM (Feb 4-5, 2017, Brussels)
○ If you have talks/activities related to ManageIQ, please email carchen@redhat.com to
coordinate ManageIQ presence and promotion
● Voting for F-release name
○ http://talk.manageiq.org/t/voting-on-name-of-f-release/1881
○ Past releases: B (13 votes), C (16 votes), D (27 votes)
○ E-release naming garnered 53 votes in total
○ F-release currently at 70 votes, with 3 more days to go. Vote vote vote!
● Metrics (past 6 months)
○ GitHub: stars 33%⬆, contributors 36%⬆, forks 37%⬆
○ Talk Forum: users 24%⬆, posts 30%⬆
○ Social media channels: 12-30%⬆

10. Classic UI
(Dan Clarizio)
● PRs merged (165)
○ Bugs (115)
○ Enhancements (16)
○ Tech Debt / Refactoring (22/11)
○ Enhancements Backported to EUWE (9)
● Bug Fixes
○ Charting/Topology/Toolbar buttons cleanup
○ Regressions from VM tree changes
○ I18n updates
○ Lots of EUWE fixes

11. Classic UI
● Tech Debt / Refactoring
○ Removal of old chart and timeline code
○ De-duplication PRs
○ Toolbars - Too many (11) to list, 25 out of 31 completed!
● Enhancements
○ Add/remove Hosts to/from Host Aggregates
○ Advanced search added to Containers
○ Containers - Label based Auto-Tagging

12. Classic UI
Add/remove Hosts to/from Host Aggregates

13. Classic UI
Advanced search added to Containers

14. Classic UI
Label based Auto-Tagging

15. Service UI
(Chris Kacerguis)
● Numerous bug fixes (~35 PRs, 18 bug fixes)
● Lots of technical debt resolved
● Added more Unit Tests (59% test coverage)
● Added yarn for dependency management
● Removed bower
● Added Power Operations in for VMs

16. Providers
(Adam Grare)

17. Hawkular
(Adam Grare)
Lead with 19 PRs merged, 18 Bugs fixed
Added a way to overwrite an existing deployment

18. RHV
(Adam Grare)
Remove full refresh when provisioning a new VM
○ Provision to a large env reduced from 17min to 1min
○ Uses the improved refresh for new VM from last sprint
VM Live Migration added
Added API to import VMs from VMware to RHEV

19. OpenStack
(Adam Grare)
Added support for new Gnocchi metrics service
Backend support for creating floating IPs using task queue

20. Amazon
(Adam Grare)
Cloud Manager refresh moved to new DTO (Data Transfer
Object) method

21. Azure
(Adam Grare)
Has a new home at
https://github.com/ManageIQ/manageiq-providers-azure

22. Lenovo
(Adam Grare)
Phase 1 of physical server refresh

23. VMware vCloud
(Adam Grare)
Credentials for vCloud Events can be added in the UI

24. Pluggable Providers
(Adam Grare)
● supports :clone
● supports :suspend
● supports :reset

25. Automate
(Greg McCullough)
Service Models
● Expose ems_events for vm_or_template
Notifications
● Added Tenant/Global notification audiences
Service
● Expose service power state

26. Automate
(Greg McCullough)
Orchestration
● Show Azure orchestration stack failure
○ Scan through its operations and delegate the error message to the stack
● Set zone when delivering a service template provision task
● Link a VM to the service through the containing orchestration stack

27. Automate
(Greg McCullough)
Fixes
● Automate schedule
○ key/pair attributes fixed to match Simulation
● MiqGroup
○ filter relationship corrected

28. Automate
(Greg McCullough)
Fixes
● Git Repository
○ Support repositories without the domain directory
■ SimpleDomain (Repository)
● SimpleDomain (Directory) (old style)
○ __domain__.yaml
■ SimpleDomain (Repository)
● __domain__.yaml
○ Refresh was not updating the contents and new references.
■ New branches and tags were not getting fetched
■ New content was not getting fetched and imported

29. Platform
(Gregg Tanzillo)
Enhancements / Bug Fixes
● Centralized Administration
● Chargeback
● Appliance
● PostgreSQL HA

30. Platform
(Gregg Tanzillo)
Centralized Administration
● Service Template Provisioning
○ ServiceOrder (shopping cart) creation on remote
region
● VM Reconfigure
○ Proper routing to remote regions
○ Create separate request per region

31. Platform
(Gregg Tanzillo)
Chargeback
● Reinstated daily rates
● Simplified rates editor
○ Disabled editing of variable rate for a fixed rate tier

32. Platform
(Gregg Tanzillo)
Appliance
● Increased default worker memory thresholds
○ Metrics processor worker: 400 to 600 MB
○ Priority worker: 400 to 600 MB
○ Generic worker: 400 to 500 MB
○ Schedule worker: 300 to 500 MB.
● Configurable number of saved logfile rotations

33. Platform
(Gregg Tanzillo)
PostgreSQL High Availability
● Raise EVM event on failover
○ “db_failover_executed”

34. API
(Alberto Bellotti)
● New Provider action import_vm which adds the ability to import VMs
○ Currently supporting importing VmWare VMs in RHEV
○ POST /api/providers/:id
{
“action” : “import_vm”,
“resource” : {
“source” : { “href” : “http://localhost:3000/api/vms/:source_vm_id” },
“target” : {
“name” : “<new_vm_name>”,
“cluster” : { “href” : “http://localhost:3000/api/clusters/:cluster_id” },
“data_store” : { “href” : “http://localhost:3000/api/data_stores/:data_store_id” },
“sparse” : true
}
}
}

35. API
(Alberto Bellotti)
● Enhanced /api/policies to support CRUD actions
POST /api/policies
{
"name" : "sample policy",
"description" : "sample policy description",
"towhat" : "ManageIQ::Providers::Redhat::InfraManager",
"conditions_ids" : [ 2, 3 ],
"policy_contents" : [{
"event_id" : 2,
"actions": [ {"action_id" : 1, "opts" : { "qualifier" : "failure" } } ]
}]
}
POST /api/policies/:id - single resource action edit
POST /api/policies - action edit for Bulk edits
POST /api/policies/:id - single resource action delete
DELETE /api/policies/:id - single resource DELETE
POST /api/policies - action delete for Bulk deletes

36. API
(Alberto Bellotti)
● Enhanced /api/conditions to support CRUD actions
POST /api/conditions
{
"name" : "sample condition",
"description" : "sample condition description",
"expression" : { "=" : { "field" : "ContainerImage-architecture", "value" : "dsa" } },
"towhat" : "ExtManagementSystem",
"modifier" : "allow"
}
POST /api/conditions/:id - single resource action edit
POST /api/conditions - action edit for Bulk edits
POST /api/conditions/:id - single resource action delete
POST /api/conditions - action delete for Bulk deletes

37. API
(Alberto Bellotti)
● New collection /api/actions with CRUD actions
Querying actions
GET /api/actions
GET /api/actions/:id
POST /api/actions - action query for Bulk queries
Creation actions via: POST /api/actions
{
"name" : "sample action",
"description" : "sample action description",
"action_type” : “custom_automation”,
"options” : {
“ae_message” : “<message>”, “ae_request” : “<request>”, “ae_hash” : { “key” : “value” }
},
}
POST /api/actions/:id - single resource action edit
POST /api/actions - action edit for Bulk edits
POST /api/actions/:id - single resource action delete
POST /api/actions - action delete for Bulk deletes

38. API
(Alberto Bellotti)
● Added ability to approve and deny /api/requests
○ When requests are not auto-approved, we now support the ability to “approve” and “deny”
requests.
POST /api/requests/:id
{
“action” : “approve”,
“resource” : { “reason” : “Reason for Request Approval” }
}
POST /api/requests/:id
{
“action” : “deny”,
“resource” : { “reason” : “Reason for Request Denial” }
}
POST /api/requests - action “approve” for bulk requests
POST /api/requests - action “deny” for bulk requests

39. API
(Alberto Bellotti)
● Id’s and other attributes now returned with Service Dialog contents.
GET /api/service_dialogs/2
{
"href": "http://localhost:3000/api/service_dialogs/2",
"id": 2,
...
"content": [
{
"id": 2,
"description": "Simple Dialog",
...
"dialog_tabs": [
{
"id": 4,
"description": "Host Info",
...
"dialog_groups": [
{
"id": 7,
"description": "Sample Box",
…
}

40. API
(Alberto Bellotti)
● Added ability to copy Service Dialogs
○ This is made available via the copy action on /api/service_dialogs/:id resources.
With this action, the Service Dialog is copied to a new Service Dialog with the label “Copy of
<current_dialog_label>” unless one is specified
POST /api/service_dialogs/:id
{
“action” : “copy”,
“resource” : { “label” : “copied_service_dialog” }
}
POST /api/service_dialogs/:id - action copy to copy a single resource
POST /api/service_dialogs - action copy to copy resources in bulk

41. API
(Alberto Bellotti)
● Added ability to delete Service Requests
○ POST /api/service_requests/:id - action delete
○ DELETE /api/service_requests/:id
○ POST /api/service_requests - action delete for Bulk deletes

42. Performance
(Dennis Metzger)
Team was focused cross-team help
manageiq-performance rubygem (Nick)
● Initial focus is on UI performance metrics
● Fundamental component for automated performance regression testing
Notable PRs (Keenan)
● Moved to Ancestry 2.2.1
● Removed an O(n^2) from save inventory
● Reduced time to load desired Resource Pools
● That Targeted Refresh thing (well refresh in general)

43. Performance
(manageiq-performance)
Collect & Report on performance metrics for MIQ
● single configurable and pluggable middleware for gathering data
● accompanying cli interface for benchmarking, reporting and analyzing
● improve automation of performance metrics
Designed for QE, Perf Team, Developers, Support
● Unify toolchain so that everyone only has to learn one
Borrows heavily from existing tools/frameworks:
● rack-mini-profiler
● cfme-performance
● perf_utils (@kbrock)

44. Performance
(manageiq-performance - middleware)
Activated by a header (WITH_PERFORMANCE_MONITORING)
Pluggable
● Request timers and info
● ActiveRecord data
● stackprof profiling (if installed)
Configurable
● Multiple storage backends
● "Browser Mode" (activate via url param / "always on")
● Configure/enable plugins

45. Performance
(manageiq-performance)
● Simplify testing of app changes
● Automate gathering and reporting of metrics
miqperf benchmark
$ bundle exec miqperf benchmark /service/explorer/s-1000000000001 --count 5
--> getting csrf_token...
--> logging in...
--> making GET request: /service/explorer/s-1000000000001
--> making GET request: /service/explorer/s-1000000000001
--> making GET request: /service/explorer/s-1000000000001
--> making GET request: /service/explorer/s-1000000000001
--> making GET request: /service/explorer/s-1000000000001
$ bundle exec miqperf benchmark --requestfile
...

46. Performance
(manageiq-performance - CLI)
Requestfile
GET => /auth_key_pair_cloud
GET => /auth_key_pair_cloud/index
GET => /auth_key_pair_cloud/show
GET => /auth_key_pair_cloud/show_list
POST => /auth_key_pair_cloud/show
POST => /auth_key_pair_cloud/show_list
GET => /availability_zone
GET => /availability_zone/index
GET => /availability_zone/show
GET => /availability_zone/show_list
POST => /availability_zone/show
POST => /availability_zone/show_list
GET => /catalog
GET => /catalog/explorer
GET => /catalog/show/1000000000001
POST => /catalog/explorer
...
$ bin/rake manageiq_performance:build_request_file

47. Performance
(manageiq-performance - CLI cont.)
miqperf report
$ bundle exec miqperf report --last
service-explorer-s-1000000000001
| ms | queries | query (ms) | rows |
| ---: | ---: | ---: | ---: |
| 4166 | 203 | 286.6 | 7065 |
| 2446 | 203 | 202.7 | 7065 |
| 2328 | 203 | 205.5 | 7065 |
| 2354 | 203 | 204.2 | 7065 |
| 2590 | 203 | 208.3 | 7065 |

48. Performance
(manageiq-performance - CLI cont.)
miqperf analyze
$ bundle exec miqperf analyze --last
tmp/manageiq_performance/run_1476917567/service%explorer%s-1000000000001
==================================
Mode: wall(1000)
Samples: 42877 (0.76% miss rate)
GC: 6207 (14.48%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
2535 (5.9%) 2535 (5.9%) Puma::Single#run
1357 (3.2%) 1338 (3.1%) block in ActiveRecord::Co...
1293 (3.0%) 1243 (2.9%) block in Sprockets::Resol...
935 (2.2%) 935 (2.2%) block in ActiveRecord::Re…
...
$ bundle exec miqperf analyze --last --method "Sprockets::Resolve#dirname_matches"

49. Performance
(manageiq-performance - TODO)
Future improvements and enhancements
● More plugins (memory, method profiling, simple timers, etc.)
● More storage backends (statsd, ELK support, etc.)
● Better reporting
● Browser user interface (for those who dislike the command line)
● MiqQueue monitoring
● IRB profile helpers (in progress...)
● Headless browser metrics
● User feedback (wink wink)

50. Performance - ancestry
(kbrock)
Ancestry ilike -> like - 120ms / same rows
count_or_objects - 720ms / ½ queries
Using scopes to access hosts - 350ms / -520 rows
Relationship arrange filters
/ems_infra/?view=show_folders
| ms |queries| query (ms)| rows | comments
| ---:| ---:| ---:| ---:| ---
| 1,917.3 | 290 | 962.9 | 1,927 | before
| 707.9 | 230 | 156.7 | 235 | after
| 63.1% | 21% | 83.7% | 88% |

51. Performance - vm/explorer
(kbrock)
vm_infra/explorer (23k vms)
Finished up tree work
Ancestry (filtering arranged), parents, resource pools
| ms |queries | query (ms) | rows | comments
| ---:| ---:| ---:| ---:| ---
| 31,738.2 | 130 | 3,981.9 | 68,205 | before
| 3,370.4 | 128 | 1,390.8 | 5,548 | after
| 89.4% | 2% | 65.1% | 92% |

52. Quality Engineering
(Dave Johnson)
○ Continuing with bug verification
○ Addressing concerns over the number of open issues
and rate of incoming bugs
○ Significant effort going into bug re-triage in an effort to
increase quality

53. Quality Engineering
(Dave Johnson)
Automation
● Seeing infinnispinny (infinite spinners) trying to find a way to catch it as it’s
sporadic
● Navmazing (new navigation framework) conversion
○ 677 -> 309 removals of old force_navigate
○ More in PRs yet to be merged
● Still a fair few locator/text changes coming in

54. Quality Engineering
(Dave Johnson)
Integration Tests 114 PRs Merged

55. Discussion
Sprint 50 Review - December 7