openstack/governance
Code Issues Proposed changes

2021 User Survey TC Question Analysis

Browse Source 
This is a follow-up to our 2019 analysis of the user
survey results.  This includes comparisons between the
2019 and 2021 date.

Change-Id: I1161093d2d728e5d4c875f6db65b0bbdcb79201b
This commit is contained in:
Jay S. Bryant
2022-04-06 16:47:58 -05:00
parent 6e4c7c5d33
commit 0cfbf9e9cd
2 changed files with 408 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/index.rst
View File

@@ -90,3 +90,4 @@ of those responses.
:titlesonly: :titlesonly:
user_survey/analysis-12-2019 user_survey/analysis-12-2019
user_survey/analysis-04-2022

407
doc/source/user_survey/analysis-04-2022.rst Normal file
View File

@@ -0,0 +1,407 @@
=====================================
Analysis of 2021 User Survey Feedback
=====================================
Introduction
------------
The OpenStack Technical Committee (TC) added questions to the User Survey for
the first time in 2019. The following questions were asked by the TC:
* How do you upgrade your version of OpenStack?
* Once on a given release, do you use stable branches for bug-fix upgrades?
* To which projects does your organization contribute maintenance resources,
such as patches for bug fixes and code reviews on master or stable
branches?
* How do members of your organization contribute to OpenStack?
* What prevents you or your organization from contributing more maintenance
resources, or makes contributing difficult?
* Other ways to participate?
The intention of these questions was to understand how users are maintaining
their OpenStack clouds and how they are interacting with the community. In the
case that they weren't currently interacting with the community it was hoped
that the questions would spark thoughts on how they could participate in the
future.
After the initial 2019 User Survey, the Technical Committee reviewed the
responses and decided that we would keep the questions the same for future
surveys with one exception. It was decided that the 'Other ways to participate'
question should be changed to a free-form field that could be completed in the case
that 'other' was selected in response to the question: 'How do members of
your organization contribute to OpenStack'.
This round of analysis will look at the results from the 2021 User Survey in
comparison to the
`2019 results <https://governance.openstack.org/tc/user_survey/analysis-12-2019.html>`_.
The 2019 User Survey result analysis is being used for comparison as
the TC did not perform an analysis of the 2020 User Survey results.
As with the previous analysis, the analysis will be performed on a per-question
basis with an over-all summary at the end.
How do you upgrade your version of OpenStack?
---------------------------------------------
As with the 2019 User Survey, this is the question that got the greatest number of
responses. Since responses to none of the questions are mandatory we need to infer the
total number that responded. There were only a couple of respondents that answered other TC
questions while leaving this answer blank. As a result, the number of responses, 276, was used
as the estimated total number of respondents to the TC's question.
+--------------------------------------------------------+-----------------+-------------------------+
| Response | Number of Users | Percentage of Responses |
+--------------------------------------------------------+-----------------+-------------------------+
| Upgrade all coordinated releases (once every 6 months) | 83 | 30 |
+--------------------------------------------------------+-----------------+-------------------------+
| Skip/Fast-forward releases | 91 | 33 |
+--------------------------------------------------------+-----------------+-------------------------+
| Not Upgrade | 65 | 23 |
+--------------------------------------------------------+-----------------+-------------------------+
| Deploy Regularly from master branch | 13 | 5 |
+--------------------------------------------------------+-----------------+-------------------------+
| Deploy all intermediary releases and all | 20 | 7 |
| coordinated releases | | |
+--------------------------------------------------------+-----------------+-------------------------+
The total number of responses for 2021 was somewhat lower than the 337 responses that
were submitted in 2019. There are a couple of positive trends, however, to notice. The first one
being that the number of 'Not Upgrade' responses decreased significantly. In 2019 35% of
respondents indicated they did not upgrade as compared to 23% in 2021. The percentage of users
that indicated that they 'Deployed Regularly from master branch' also decreased from 9% to 5%
while 'Deploy all intermediary releases and all coordinated releases' stayed constant at 7%.
The increase in users upgrading their releases appears to be pretty evenly spread, percentage wise,
between 'Upgrade all coordinated releases (once every 6 months)' and 'Skip/Fast-forward releases'
increasing from 22% to 30% and 27% to 33% respectively. Unfortunately, we don't know that the
respondents that no longer indicated that they did not upgrade are the same ones that now indicate
that they do upgrade. With that said, the decrease does appear to be a positive trend. It will be
worth watching these results in subsequent surveys to see if that trend continues.
One other data point worth noting is the fact that we have reached a point where the number of
users using the 'Upgrade all coordinated releases' and 'Skip/Fast-forward releases' options
are comparable. The TC sees this as an indication that the changes we have made in the
release process to allow doing Skip/Fast-forward upgrades have been worth the effort that
was required. As we continue to improve/refine the upgrade options these are data points
that we will want to continue to watch.
Once on a given release, do you use stable branches for bug-fix upgrades?
-------------------------------------------------------------------------
Consistent with the 2019 survey, the number of responses to this question was high, at 265 responses
and had a notable total increase from 216 responses in 2019.
+-------------------------------------------------------------+-------+-------------------------+
| Answer | Users | Percentage of Responses |
+-------------------------------------------------------------+-------+-------------------------+
| I do not do bug-fix upgrades | 46 | 17 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, backporting specific fixes | 44 | 17 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, deploying every commit on the stable branch | 18 | 7 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, upgrading at various points in time depending on fixes | 75 | 28 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, using only official point releases | 79 | 30 |
+-------------------------------------------------------------+-------+-------------------------+
The responses in 2021 were very consistent with the 2019 results. The number of users reporting
that they do not do bug-fix upgrades or backport specific fixes decreased from 19% and
21% respectively. The number of respondents that indicate that they only deploy official point
releases also decreased slightly from 31% to 30%.
While the number of users to indicate that they deploy every commit remained the least reported
approach, it did increase from 4% in 2019 to 7% in 2021. Users indicating that they upgrade
depending on the fixes available increased from 26% to 28%.
So, overall, the results for this question in the 2021 survey were consistent with those collected in 2019.
To which projects does your organization contribute maintenance resources such as patches for bug and reviews on master or stable branches?
-------------------------------------------------------------------------------------------------------------------------------------------
This question had a good increase in responses compared to 2019, with only about 33% of users
responding previously compared to 128, or 46%, in 2021. As in 2019, the majority of contributions
are made to `core projects
<https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html#openstack-service-overview>`_ .
There is, however, a new entry in the top five projects: Kolla.
`Kolla <https://wiki.openstack.org/wiki/Kolla>`_ is an OpenStack community project that provides
production ready containers and deployment tools for operating OpenStack clouds. Given the
continued growth in popularity of containerizing workloads, the increase in contributors is
not particularly surprising and would seem to be a good sign for OpenStack as users are seeking
newer and better ways to deploy OpenStack clouds.
All of the projects in the top 5 showed a notable increase in reported participation from the
2019 survey. While Glance was knocked out of the Top 5 projects, it still had a reported
increase from 25 respondents indicating activity in the project to 27.
+----------+--------------+
| Project | Contributors |
+----------+--------------+
| Nova | 63 |
+----------+--------------+
| Neutron | 56 |
+----------+--------------+
| Cinder | 40 |
+----------+--------------+
| Keystone | 34 |
+----------+--------------+
| Kolla | 34 |
+----------+--------------+
After the first review of these results in the 2019 user survey,
the TC wanted to compare the number of respondents who reported they
used a project as compared to the number who reported that they contribute
to the project. For the 2021 results review we have continued this analysis
but have also added the previous percentage of participation for
reference. To get this information we considered the number of
users who reported that they were using the project in production.
Note that we did not count users who indicated that they were just
testing a service or who indicated that they had the service
installed as part of a Proof of Concept. So the number of users
that are using any given service may be notably larger than
indicated in the results below. Here are the results of
that investigation.
+-------------------+--------------+-------+-----------------+----------------------+
| Project | Contributors | Users | % Participation | 2019 % Participation |
+-------------------+--------------+-------+-----------------+----------------------+
| Aodh | 3 | 31 | 9 | 25 |
+-------------------+--------------+-------+-----------------+----------------------+
| Barbican | 8 | 70 | 11 | 19 |
+-------------------+--------------+-------+-----------------+----------------------+
| Blazar | 6 | 4 | 150 | 100 |
+-------------------+--------------+-------+-----------------+----------------------+
| Ceilometer | 10 | 84 | 12 | 14 |
+-------------------+--------------+-------+-----------------+----------------------+
| Cinder | 40 | 216 | 19 | 9 |
+-------------------+--------------+-------+-----------------+----------------------+
| Cloudkitty | 4 | 12 | 33 | 23 |
+-------------------+--------------+-------+-----------------+----------------------+
| Cyborg | 3 | 4 | 75 | 100 |
+-------------------+--------------+-------+-----------------+----------------------+
| Designate | 13 | 55 | 24 | 25 |
+-------------------+--------------+-------+-----------------+----------------------+
| Glance | 27 | 231 | 12 | 8 |
+-------------------+--------------+-------+-----------------+----------------------+
| Heat | 13 | 164 | 8 | 8 |
+-------------------+--------------+-------+-----------------+----------------------+
| Horizon | 18 | 224 | 8 | 6 |
+-------------------+--------------+-------+-----------------+----------------------+
| Ironic | 23 | 65 | 35 | 27 |
+-------------------+--------------+-------+-----------------+----------------------+
| Keystone | 34 | 235 | 15 | 9 |
+-------------------+--------------+-------+-----------------+----------------------+
| Kolla | 34 | 69 | 49 | 52 |
+-------------------+--------------+-------+-----------------+----------------------+
| Kuryr | 5 | 8 | 63 | 71 |
+-------------------+--------------+-------+-----------------+----------------------+
| LOCI | 3 | 7 | 43 | 40 |
+-------------------+--------------+-------+-----------------+----------------------+
| Magnum | 13 | 42 | 31 | 29 |
+-------------------+--------------+-------+-----------------+----------------------+
| Manila | 13 | 26 | 50 | 23 |
+-------------------+--------------+-------+-----------------+----------------------+
| Masakari | 2 | 14 | 14 | 17 |
+-------------------+--------------+-------+-----------------+----------------------+
| Mistral | 8 | 23 | 35 | 35 |
+-------------------+--------------+-------+-----------------+----------------------+
| Monasca | 4 | 33 | 12 | 23 |
+-------------------+--------------+-------+-----------------+----------------------+
| Murano | 3 | 6 | 50 | 18 |
+-------------------+--------------+-------+-----------------+----------------------+
| Neutron | 56 | 229 | 24 | 15 |
+-------------------+--------------+-------+-----------------+----------------------+
| Nova | 63 | 230 | 27 | 15 |
+-------------------+--------------+-------+-----------------+----------------------+
| Octavia | 26 | 89 | 29 | 35 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack Client | 21 | 191 | 11 | 8 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack Ansible | 27 | 59 | 46 | 38 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack Helm | 5 | 14 | 36 | 23 |
+-------------------+--------------+-------+-----------------+----------------------+
| Panko | 3 | 7 | 43 | 45 |
+-------------------+--------------+-------+-----------------+----------------------+
| QA | 2 | N/A | N/A | N/A |
+-------------------+--------------+-------+-----------------+----------------------+
| Rally | 6 | 44 | 14 | 16 |
+-------------------+--------------+-------+-----------------+----------------------+
| Sahara | 1 | 2 | 50 | 21 |
+-------------------+--------------+-------+-----------------+----------------------+
| Senlin | 4 | 5 | 80 | N/A |
+-------------------+--------------+-------+-----------------+----------------------+
| Swift | 19 | 102 | 19 | 11 |
+-------------------+--------------+-------+-----------------+----------------------+
| Tacker | 1 | 3 | 33 | 75 |
+-------------------+--------------+-------+-----------------+----------------------+
| Trove | 4 | 14 | 29 | 15 |
+-------------------+--------------+-------+-----------------+----------------------+
| TripleO | 11 | 26 | 42 | 16 |
+-------------------+--------------+-------+-----------------+----------------------+
| Watcher | 2 | 6 | 33 | N/A |
+-------------------+--------------+-------+-----------------+----------------------+
| Zaqar | 2 | 9 | 22 | 23 |
+-------------------+--------------+-------+-----------------+----------------------+
| Zun | 2 | 8 | 25 | N/A |
+-------------------+--------------+-------+-----------------+----------------------+
A couple of things to note from the results above. The numbers seem to
confirm one concern that has been previously raised within the community.
There appear to be a number of the smaller projects that are being used
and maintained by a small group of OpenStack users. It is also interesting
the disparity between the number of users and contributors for the core
projects like Cinder, Glance, Horizon and Keystone. It is unclear from
these results as to whether the disparity is because these projects
have become more stable and don't require as much development or
if there are other reasons for this disparity.
When considering the percentage of participation as compared to the 2019
User Survey results there are a number of things to note. First, there
has not been a notable decrease in user participation across projects.
Also, it is interesting to note that a majority of projects continued to
have consistent numbers as compared to the previous survey. A number of projects
even showed an increase in participation. This is true for many of the
core projects like Cinder, Keystone, Glance, Ironic, Neutron and Nova.
We don't know if these changes are due to a change in the demographics of
the respondents (more user/developers responding vs. just users) or if it
is a positive indication in the health of the community. Overall, a key take
away would be that project participation has remained largely stable or grown
slightly since the last user survey.
How do members of your organization contribute to OpenStack?
------------------------------------------------------------
As with the other questions in the user survey, this question also saw an increase
in responses. 173 users responded for a total of 63% of the total.
+----------------------------------------------+-------+-------------------------+
| Contribution | Users | Percentage of Responses |
+----------------------------------------------+-------+-------------------------+
| Bug reports | 173 | 100 |
+----------------------------------------------+-------+-------------------------+
| Bug fixes on master | 85 | 49 |
+----------------------------------------------+-------+-------------------------+
| Participate in forum sessions at the summit | 59 | 34 |
+----------------------------------------------+-------+-------------------------+
| Participate in PTG sessions | 44 | 25 |
+----------------------------------------------+-------+-------------------------+
| Code review on master | 43 | 25 |
+----------------------------------------------+-------+-------------------------+
| Particpate in ops meetups | 43 | 25 |
+----------------------------------------------+-------+-------------------------+
| Documentation improvement | 39 | 23 |
+----------------------------------------------+-------+-------------------------+
| Backporting bug fixes to stable branches | 37 | 21 |
+----------------------------------------------+-------+-------------------------+
| Sponsor in-person events | 28 | 16 |
+----------------------------------------------+-------+-------------------------+
| Code review on stable branches | 26 | 15 |
+----------------------------------------------+-------+-------------------------+
| Feature design review | 22 | 13 |
+----------------------------------------------+-------+-------------------------+
| Contribute resources to run CI jobs upstream | 8 | 5 |
+----------------------------------------------+-------+-------------------------+
| Host third-party CI jobs downstream | 4 | 2 |
+----------------------------------------------+-------+-------------------------+
In the 2019 survey the TC noted that the responses are very interesting given that
this is a 'User Survey'. The responses show that OpenStack is unique in the fact that
it is a user driven community with many of the users also being contributors. In fact
we had more respondents indicating that they contributed to OpenStack than in 2019.
A couple of things of interest to note. First, the number of respondents that indicated
they contributed bug reports increased notably from 86% in 2019 to 100% in 2021. More
importantly, the number of users that indicated that they contribute bug fixes on master
increased from 36% to 49%. This takes this response from 4th most popular way of
contributing to 2nd!
Some of the decreased numbers in participation are likely to be due to the pandemic that
we all have been dealing with. Reported participation in summit sessions and ops meetups
both decreased from 47% to 34% and 39% to 25% respectively. Similarly, sponsorship of
in-person events dropped from 20% to 16%. These drops, however, are not as significant as
expected given the pandemic, which leads to TC to question these results. For future surveys
we may want to look at the wording of the questions. Perhaps users are responding based
upon all previous activity, not just the activity since the last user survey.
A decrease in participation that could have been anticipated by many of the community
members is in code reviews for both master and stable branches. The number of users indicating
participating in these activities decreased by 6% for master code reviews and 5% for stable
branch reviews. This is consistent with what has been seen in many of the projects. Code is
being submitted but the time required to get patches reviewed and merged has increased. This
is also contributed to by a notable decrease in participants hosting third-party and CI job
resources.
The rest of the participation responses were relatively consistent with what was reported
in the 2019 User Survey.
What prevents you or your organization from contributing more maintenance resources, or makes contributing difficult?
---------------------------------------------------------------------------------------------------------------------
This question is the one that had the most notable increase in
responses from 2019. Only 19% of users (69) responded to this
question previously as compared to 47% (131) in 2021. This could
be taken in a positive light, that more users are thinking
about contributing. Unfortunately, the challenges continue to
be similar.
Some combination of time, resources and company support accounted
for more than half of the responses. Not having the appropriate skills
to contribute was also a common theme in the responses. The TC might
be able to help with this by raising awareness of the OpenStack Jobs
board.
A few responses indicated that they felt that they were not able to
contribute because the version of OpenStack that they are on was
so far behind the current release. This response seems somewhat
surprising as problems found in the older releases may still need to
be fixed in the current release. In some cases, it is possible that
the code will have changed significantly from the release that is running.
With that said, this is a response that probably deserves further
discussion by the TC.
The other response that was repeated by at least a few users was
communication and timezone barriers. This is a challenge that has
been known by the TC and OpenInfra Foundation for some time. Changes
have been made to try to alleviate some of this problem but it
sounds like it hasn't worked for everyone.
Other ways users participate:
-----------------------------
This was the field that we changed the wording for from the 2019 user
survey. There were only two responses submitted: one listing bug reports
again and one indicating that they work with vendors to submit requests
for enhancement.
So, this field is not getting a lot of input. Worth discussion among the
TC as to whether this should be removed or replaced with a different
question.
Summary
-------
In summary, the questions being asked by the TC in the User Survey are providing
useful information. While it was unfortunate to see the total number of
responses decrease, the fact that we had higher participation responding to the
TC's questions is encouraging. Hopefully this trend will continue as the numbers
are more meaningful as we have more years of data for comparison.
It is also encouraging to see the numbers moving in the desired direction for
a number of the questions that we are asking. The responses to how/when users are
upgrading their OpenStack clouds seems to suggest that the efforts of the TC and
the community as a whole have had a positive impact on the user experience, or at
least their upgrade habits. The overall improvement in user engagement, participating
in the community is also encouraging. The data suggests that the OpenStack
community continues to be a consistently active one.
All-in-all, the TC is pleased with the continued responses we are getting and
would like to continue to collect this data for future analysis.
Additional Resources
--------------------
The `OpenStack Survey Report
<https://www.openstack.org/analytics>`_ also provides
a graphical overview of the OpenStack Survey
results.