openstack/governance
Code Issues Proposed changes

Merge "2022 User Survey TC Question Analysis"

Browse Source
This commit is contained in:
Zuul 2023-11-07 20:33:08 +00:00 committed by Gerrit Code Review
commit 85054cddfe
3 changed files with 429 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
doc/source/index.rst
View File

@ -81,9 +81,27 @@ exceptions are listed in the :doc:`reference/house-rules` document.
Summary of User Survey Questions Responses
==========================================
Since 2019 the OpenStack Technical Committee has added questions to the
annual OpenStack User Survey. Below are links to summaries and analysis
of those responses.
Since 2019 the OpenStack Technical Committee has added questions to the annual
OpenStack User Survey. In 2019, 2021 and 2022 survey the following questions
were asked by the TC:
* How do you upgrade your version of OpenStack?
* Once on a given release, do you use stable branches for bug-fix upgrades?
* To which projects does your organization contribute maintenance resources,
such as patches for bug fixes and code reviews on master or stable
branches?
* How do members of your organization contribute to OpenStack?
* What prevents you or your organization from contributing more maintenance
resources, or makes contributing difficult?
* Other ways to participate?
The intention of these questions was to understand how users are maintaining
their OpenStack clouds and how they are interacting with the community. In the
case that they weren't currently interacting with the community it was hoped
that the questions would spark thoughts on how they could participate in the
future.
Below are links to summaries and analysis of those responses.
.. toctree::
:maxdepth: 2
@ -91,3 +109,4 @@ of those responses.
user_survey/analysis-2019
user_survey/analysis-2021
user_survey/analysis-2022

31
doc/source/user_survey/analysis-2021.rst
View File

@ -5,35 +5,8 @@ Analysis of 2021 User Survey Feedback
Introduction
------------
The OpenStack Technical Committee (TC) added questions to the User Survey for
the first time in 2019. The following questions were asked by the TC:
* How do you upgrade your version of OpenStack?
* Once on a given release, do you use stable branches for bug-fix upgrades?
* To which projects does your organization contribute maintenance resources,
such as patches for bug fixes and code reviews on master or stable
branches?
* How do members of your organization contribute to OpenStack?
* What prevents you or your organization from contributing more maintenance
resources, or makes contributing difficult?
* Other ways to participate?
The intention of these questions was to understand how users are maintaining
their OpenStack clouds and how they are interacting with the community. In the
case that they weren't currently interacting with the community it was hoped
that the questions would spark thoughts on how they could participate in the
future.
After the initial 2019 User Survey, the Technical Committee reviewed the
responses and decided that we would keep the questions the same for future
surveys with one exception. It was decided that the 'Other ways to participate'
question should be changed to a free-form field that could be completed in the case
that 'other' was selected in response to the question: 'How do members of
your organization contribute to OpenStack'.
This round of analysis will look at the results from the 2021 User Survey in
comparison to the
`2019 results <https://governance.openstack.org/tc/user_survey/analysis-12-2019.html>`_.
This is an analysis of the results from the 2021 User Survey in comparison to
the `2019 results <https://governance.openstack.org/tc/user_survey/analysis-2019.html>`_.
The 2019 User Survey result analysis is being used for comparison as
the TC did not perform an analysis of the 2020 User Survey results.
As with the previous analysis, the analysis will be performed on a per-question

405
doc/source/user_survey/analysis-2022.rst Normal file
View File

@ -0,0 +1,405 @@
=====================================
Analysis of 2022 User Survey Feedback
=====================================
Introduction
------------
This is an analysis of the results from the 2022 User Survey in comparison to
the `2021 results <https://governance.openstack.org/tc/user_survey/analysis-2021.html>`_.
As with the previous analysis, the analysis will be performed on a per-question
basis with an over-all summary at the end.
How do you upgrade your version of OpenStack?
---------------------------------------------
+---------------------------------------------------------------+-------+-------------------------+
| Response | Users | Percentage of Responses |
+---------------------------------------------------------------+-------+-------------------------+
| Skip/fast-forward releases | 84 | 34 |
+---------------------------------------------------------------+-------+-------------------------+
| Upgrade all coordinated releases (once every 6 months) | 75 | 30 |
+---------------------------------------------------------------+-------+-------------------------+
| Not upgrade | 65 | 26 |
+---------------------------------------------------------------+-------+-------------------------+
| Deploy regularly from the master branch | 12 | 4 |
+---------------------------------------------------------------+-------+-------------------------+
| Deploy all intermediary releases and all coordinated releases | 12 | 4 |
+---------------------------------------------------------------+-------+-------------------------+
The total number of responses was slightly lower than in the 2021 survey - 272
in 2021 and 249 in 2022.
Results in this year's survey are almost the same as from last year. The only
differences are a small increase of users who do not do upgrades and in the same
time decrease amount of users who deploy all intermediary releases and all
coordinated releases. But as differences are small (2-3%) this doesn't show any
significant change.
It is very interesting how results of this question will change (or not) in next
year's survey, when SLURP releases will be available.
Once on a given release, do you use stable branches for bug-fix upgrades?
-------------------------------------------------------------------------
Number of responses for this question was a bit smaller than in 2021, similar
to the previous questions. There were 235 responses in 2022 compared to the 265
in the 2021 survey.
+-------------------------------------------------------------+-------+-------------------------+
| Response | Users | Percentage of Responses |
+-------------------------------------------------------------+-------+-------------------------+
| I do not do bugfix upgrades | 39 | 17 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, backporting specific fixes | 50 | 21 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, deploying every commit on the stable branch | 16 | 7 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, upgrading at various points in time depending on fixes | 71 | 30 |
+-------------------------------------------------------------+-------+-------------------------+
| Yes, using only official point releases | 59 | 25 |
+-------------------------------------------------------------+-------+-------------------------+
The responses for the question about stable branches updates are very consistent
with 2021 survey. There are no major changes in how results are distributed. The
only change is an increase in the number of users who are backporting specific
fixes (21% this year vs. 17% in 2021) and a decrease in the number of users who
are using only official point releases (25% this year vs. 30% in 2021). This
shows that there is a significant number of users who are backporting fixes
selectively on their own rather than waiting for official point releases which
will include required fixes.
To which projects does your organization contribute maintenance resources such as patches for bug and reviews on master or stable branches?
-------------------------------------------------------------------------------------------------------------------------------------------
Overall participation level for this questions is similar to the 2021 survey -
about 44% of users who provided information about used projects, participated
also in this question.
In the TOP 5 projects according to the number of contributors there is one
change - `Ironic <https://wiki.openstack.org/wiki/Ironic>`_ is now listed as
5th project with top number of contributors. It replaced `Kolla
<https://wiki.openstack.org/wiki/Kolla>`_ which was ranked 5th in the 2021
survey.
Majority of the projects (3/5) showed minor decrease in reported number of
contributors from 2021.
+----------+--------------+----------------------+
| Project | Contributors | Contributors in 2021 |
+----------+--------------+----------------------+
| Nova | 63 | 63 |
+----------+--------------+----------------------+
| Neutron | 53 | 56 |
+----------+--------------+----------------------+
| Cinder | 31 | 40 |
+----------+--------------+----------------------+
| Keystone | 30 | 34 |
+----------+--------------+----------------------+
| Ironic | 25 | 23 |
+----------+--------------+----------------------+
After the first review of these results in the 2019 user survey,
the TC wanted to compare the number of respondents who reported they
used a project as compared to the number who reported that they contribute
to the project.
For the 2021 results review we have continued the analysis to see how many of
respondents who reported they used a project also contribute to the project. In
2022 we have continued that but have also added the previous percentage of
participation for reference. To get this information we considered the number of
users who reported that they were using the project in production.
Note that we did not count users who indicated that they were just
testing a service or who indicated that they had the service
installed as part of a Proof of Concept. So the number of users that are using
any given service may be notably larger than indicated in the results below.
Here are the results of that investigation.
+-------------------+--------------+-------+-----------------+----------------------+
| Project | Contributors | Users | % Participation | 2021 % Participation |
+-------------------+--------------+-------+-----------------+----------------------+
| AODH | 3 | 28 | 11 | 9 |
+-------------------+--------------+-------+-----------------+----------------------+
| Barbican | 10 | 81 | 12 | 11 |
+-------------------+--------------+-------+-----------------+----------------------+
| Blazar | 3 | 5 | 60 | 150 |
+-------------------+--------------+-------+-----------------+----------------------+
| Ceilometer | 10 | 75 | 13 | 12 |
+-------------------+--------------+-------+-----------------+----------------------+
| Cinder | 31 | 213 | 15 | 19 |
+-------------------+--------------+-------+-----------------+----------------------+
| Cloudkitty | 5 | 12 | 42 | 33 |
+-------------------+--------------+-------+-----------------+----------------------+
| Congress | - | 2 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Cyborg | 2 | 7 | 29 | 75 |
+-------------------+--------------+-------+-----------------+----------------------+
| Designate | 14 | 53 | 26 | 24 |
+-------------------+--------------+-------+-----------------+----------------------+
| Freezer | - | 2 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Glance | 21 | 218 | 10 | 12 |
+-------------------+--------------+-------+-----------------+----------------------+
| Heat | 16 | 154 | 10 | 8 |
+-------------------+--------------+-------+-----------------+----------------------+
| Horizon | 18 | 199 | 9 | 8 |
+-------------------+--------------+-------+-----------------+----------------------+
| Ironic | 25 | 64 | 39 | 35 |
+-------------------+--------------+-------+-----------------+----------------------+
| Keystone | 30 | 226 | 13 | 15 |
+-------------------+--------------+-------+-----------------+----------------------+
| Kolla | 18 | 59 | 31 | 49 |
+-------------------+--------------+-------+-----------------+----------------------+
| Kuryr | 2 | 5 | 40 | 63 |
+-------------------+--------------+-------+-----------------+----------------------+
| LOCI | 3 | 6 | 50 | 43 |
+-------------------+--------------+-------+-----------------+----------------------+
| Magnum | 10 | 50 | 20 | 42 |
+-------------------+--------------+-------+-----------------+----------------------+
| Masakari | 5 | 8 | 63 | 14 |
+-------------------+--------------+-------+-----------------+----------------------+
| Manila | 10 | 36 | 28 | 50 |
+-------------------+--------------+-------+-----------------+----------------------+
| Mistral | 6 | 16 | 38 | 35 |
+-------------------+--------------+-------+-----------------+----------------------+
| Monasca | - | 5 | 0 | 12 |
+-------------------+--------------+-------+-----------------+----------------------+
| Murano | 1 | 8 | 13 | 50 |
+-------------------+--------------+-------+-----------------+----------------------+
| Neutron | 53 | 216 | 25 | 24 |
+-------------------+--------------+-------+-----------------+----------------------+
| Nova | 63 | 217 | 29 | 27 |
+-------------------+--------------+-------+-----------------+----------------------+
| Octavia | 21 | 112 | 19 | 29 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack-Ansible | 23 | 58 | 40 | 46 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack-Helm | 4 | 8 | 50 | 36 |
+-------------------+--------------+-------+-----------------+----------------------+
| OpenStack Client | 19 | 179 | 11 | 11 |
+-------------------+--------------+-------+-----------------+----------------------+
| Panko | - | 10 | 0 | 43 |
+-------------------+--------------+-------+-----------------+----------------------+
| Qinling | - | 0 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Rally | 5 | 38 | 13 | 14 |
+-------------------+--------------+-------+-----------------+----------------------+
| Sahara | 1 | 2 | 50 | 50 |
+-------------------+--------------+-------+-----------------+----------------------+
| Searchlight | - | 1 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Senlin | 4 | 4 | 100 | 80 |
+-------------------+--------------+-------+-----------------+----------------------+
| Solum | - | 0 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Storlets | - | 0 | 0 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Swift | 18 | 90 | 20 | 19 |
+-------------------+--------------+-------+-----------------+----------------------+
| Tacker | 1 | 1 | 100 | 33 |
+-------------------+--------------+-------+-----------------+----------------------+
| Tricircle | 1 | 1 | 100 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| TripleO | 3 | 23 | 13 | 42 |
+-------------------+--------------+-------+-----------------+----------------------+
| Trove | 6 | 9 | 67 | 29 |
+-------------------+--------------+-------+-----------------+----------------------+
| Vitrage | 2 | 2 | 100 | - |
+-------------------+--------------+-------+-----------------+----------------------+
| Watcher | 2 | 3 | 67 | 33 |
+-------------------+--------------+-------+-----------------+----------------------+
| Zaqar | - | 5 | 0 | 22 |
+-------------------+--------------+-------+-----------------+----------------------+
| Zun | - | 4 | 0 | 25 |
+-------------------+--------------+-------+-----------------+----------------------+
There are few things worth to note regarding the list above.
Projects like: Monasca, Panko, Zaqar and Zun still have users who use it in the
Production but they lost all of their contributors.
Those projects joined the group of the projects with 0 declared contributors
which includes also: Congress, Freezer, Solum and Storlets.
Couple of projects like TripleO, Murano or Cyborg have experienced significant
drop in the participation level in 2022 comparing to the previous year.
The good thing to note is that none of the core projects like e.g. Cinder,
Keystone, Glance, Ironic, Neutron, Nova, Horizon, OpenStack Client have
experienced contributors drop.
How do members of your organization contribute to OpenStack?
------------------------------------------------------------
As with the other questions in the user survey, this question also saw an
decrease in responses. 111 users responded to it in this years' survey
comparing to the 173 users last year.
+----------------------------------------------+-------+-------------------------+
| Response | Users | Percentage of Responses |
+----------------------------------------------+-------+-------------------------+
| Bug reports | 93 | 84 |
+----------------------------------------------+-------+-------------------------+
| Documentation improvement | 47 | 42 |
+----------------------------------------------+-------+-------------------------+
| Participate in Forum sessions at the Summit | 44 | 40 |
+----------------------------------------------+-------+-------------------------+
| Bug fixes on master | 43 | 39 |
+----------------------------------------------+-------+-------------------------+
| Participate in Ops meetups | 37 | 33 |
+----------------------------------------------+-------+-------------------------+
| Participate in PTG sessions | 30 | 27 |
+----------------------------------------------+-------+-------------------------+
| Code review on master | 27 | 24 |
+----------------------------------------------+-------+-------------------------+
| Backporting bug fixes to stable branches | 25 | 23 |
+----------------------------------------------+-------+-------------------------+
| Sponsor in-person events | 24 | 22 |
+----------------------------------------------+-------+-------------------------+
| Code review on stable branches | 24 | 22 |
+----------------------------------------------+-------+-------------------------+
| Feature development | 23 | 21 |
+----------------------------------------------+-------+-------------------------+
| Feature design review | 14 | 13 |
+----------------------------------------------+-------+-------------------------+
| Contribute resources to run CI jobs upstream | 8 | 7 |
+----------------------------------------------+-------+-------------------------+
| Host third-party CI jobs downstream | 7 | 6 |
+----------------------------------------------+-------+-------------------------+
| Other | 1 | 1 |
+----------------------------------------------+-------+-------------------------+
Similarly to the last year results, most people declare that they report bugs
as their way of contribution to the OpenStack project.
Very good thing to note is the fact that "Documentation improvement" response
have got significant increase of the users (47 users this year vs. 39 in the
2021).
From the other hand different ways of the participation in the events decreased
significantly. One of the potential reasons for that for the in person events
may be effect of the overall global economic situation where many companies have
experienced budget difficulties and are looking for a ways how to save some
money. Other potential reasons for that may be for example:
* health concern,
* environmental concerns,
* political issues.
This is just a speculation as there was no question about this in the survey.
If we want to know more details about what prevents people from participating in
the in person events, We should consider adding new question to the next user
survey.
What prevents you or your organization from contributing more maintenance resources, or makes contributing difficult?
---------------------------------------------------------------------------------------------------------------------
There were 101 users who provided response for this question in the 2022 survey.
It is a decrease of the number of responses comparing to the 2021 where 131
users answers to this question.
+--------------------------------------------------+-------+-------------------------+
| Response | Users | Percentage of Responses |
+--------------------------------------------------+-------+-------------------------+
| Lack of resources | 54 | 53 |
+--------------------------------------------------+-------+-------------------------+
| Lack of skills | 14 | 14 |
+--------------------------------------------------+-------+-------------------------+
| Bug tracker tool is not intuitive | 11 | 11 |
+--------------------------------------------------+-------+-------------------------+
| Inconvient way of receiving patches | 11 | 11 |
+--------------------------------------------------+-------+-------------------------+
| Difficult process | 8 | 8 |
+--------------------------------------------------+-------+-------------------------+
| Business decission | 3 | 3 |
+--------------------------------------------------+-------+-------------------------+
| Using too old OpenStack version | 3 | 3 |
+--------------------------------------------------+-------+-------------------------+
| No need to contribute as it just works as needed | 3 | 3 |
+--------------------------------------------------+-------+-------------------------+
| Paying vendor for support | 2 | 2 |
+--------------------------------------------------+-------+-------------------------+
| Specific changes, rejected by the community | 2 | 2 |
+--------------------------------------------------+-------+-------------------------+
| Language barrier | 2 | 2 |
+--------------------------------------------------+-------+-------------------------+
| Security; Private Network | 1 | 1 |
+--------------------------------------------------+-------+-------------------------+
| Company focus shifted to other projects | 1 | 1 |
+--------------------------------------------------+-------+-------------------------+
| Difficult to get started | 1 | 1 |
+--------------------------------------------------+-------+-------------------------+
The most common reason which prevents organizations from contributing is lack of
resources.
The second from the top, also repeated many times was lack of skills. There was
also one user who responded that it's "difficult to start" with them
which could actually be included in the same category.
There was 8 people who replied that the main reason which stops it from
contributing is generally "difficult process". It's worth to look deeper into
this category of replies and see in detail what were the answers there:
+--------------------------------------------------------------------------------+
| Most problems we encounter are in the gaps between projects which means |
| neither project team will take responsibility and review/help out. |
| "Drive by" patches from deployers are often one liners and attract -1 |
| reviews on the basis of "theres no testing of this whole area in |
| devstack- so this is -1". |
| Operators have little or no interest in devstack and and not in a position |
| to fix it's technical debt. |
+--------------------------------------------------------------------------------+
| Some projects have major lead times between patch contributions and reviews, |
| with many picky reviews on non-critical areas. |
| Thus we are not able to contribute significantly to Nova for example, |
| unless needed because the staff get frustrated with the lack of progress |
| and are on short contracts. |
+--------------------------------------------------------------------------------+
| Cumbersome upstream review processes make contributing fixes a time and thus |
| cost intensive undertaking - this is the reason none of our drivers was ever |
| tried to put to upstream but rather made public ourselfes. |
+--------------------------------------------------------------------------------+
| Internal struggle with OS policies. |
+--------------------------------------------------------------------------------+
| scope creep in bugfixes/regressions (i.e. "please make your re-added feature |
| which fixes a regression generally usable for ipv4 and ipv6" when the original |
| solution- which was implemented upstream- didn't implement ipv6 at all) |
+--------------------------------------------------------------------------------+
| The time to have a fix "production ready" for upstream. |
+--------------------------------------------------------------------------------+
| The onboarding and contribution process is pretty high-bar (custom |
| infrastructure compared to e.g. gitlab.com/github.com, gerrit workflow is |
| unfamiliar to many); Oftentimes the issues we encounter are already fixed |
| upstream. |
+--------------------------------------------------------------------------------+
From above replies it seems that main reason why users see our process as
difficult is generally review process and requirement for high quality patches,
with good testing coverage when many operators just wants to have their issue
fixed fast.
Other interesting point is 3 users who replied that they don't need to
contribute as OpenStack works for them as expected.
Other ways users participate:
-----------------------------
There were no users responses to this question this year. It's very similar to
what was in the 2021 survey where we had just 2 responses to the same question.
Summary
-------
Unfortunately we noticed decrease of the overall number of users responding to
the TC questions in 2022 user survey comparing to the 2021 survey. The good
thing is that there is still a significant number of responses so we can get
a lot of useful information from it.
After 2021 survey numbers in most cases are very similar in the 2022 survey.
It seems that things like, how users are upgrading OpenStack, what maintenance
releases they are using or how people are contributing to the OpenStack in
general is stable.
The fact that there is still increasing number of projects with lack of
contributors is alarming and TC should probably thing more about actions
possible to take to address it.
Additional Resources
--------------------
The `OpenStack Survey Report
<https://www.openstack.org/analytics>`_ also provides
a graphical overview of the OpenStack Survey
results.