1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
-*- mode: org -*-
This service is intended to assist with quality assurance within
Guix. That is maintaining and improving the quality of many aspects of
Guix, like packages for example.
* Local Development
Use the guix-dev.scm file to provide the dependencies.
#+BEGIN_SRC shell
guix environment -l guix-dev.scm
#+END_SRC
Then run the following commands:
#+BEGIN_SRC shell
./bootstrap.sh
./configure
make
#+END_SRC
After that, you can start the service via:
#+BEGIN_SRC shell
./pre-inst-env scripts/guix-qa-frontpage
#+END_SRC
Expect pages to load slowly since the QA frontpage won't have up to date
cached data to use.
* Information flow
#+CAPTION: Diagram describing how information moves between different components
[[./qa-information-flow.png]]
* Ideas
** For branches
*** TODO Show broken packages
*** TODO Show broken system tests
**** TODO Submit builds for branch system tests
*** TODO Show broken fixed output package derivations
*** TODO Show lint warnings
*** Branch comparisons
**** TODO Show broken system tests
**** TODO Show broken fixed output package derivations
**** TODO Show new lint warnings
*** DONE Show package reproducibility statistics
This will provide a better URL and faster page load times compared to
directly going to data.guix.gnu.org or data.qa.guix.gnu.org.
** For patches
*** TODO Better handle an influx of new bugs
Currently when this happens, I think QA may delete a bunch of branches that
now fall outside the latest issues it looks at, only to re-process these once
they reappear as the new issues are dealt with (either by being merged or
closed).
*** TODO Show and describe the overall status on the page for an issue
*** TODO Improve display of lint warning changes
**** TODO Say which package the lint warnings apply to
*** TODO Show when there are comments/messages on the issue
To highlight when there is discussion that might need reading before
merging the patch.
*** TODO Show changes with system tests
**** TODO Submit builds for system tests when affected by patches
*** TODO Extend testing for patches
**** TODO Cross-compilation from =x86_64-linux=
This can be enabled, but data might often be lacking due to lack of build
resources for bordeaux.guix.gnu.org
**** TODO =powerpc64le-linux=
The one powerpc64le-linux build machine for bordeaux.guix.gnu.org isn't always
on (this could be fixed by changing the hosting). Anyway, some always
available build machine for powerpc64le-linux is probably needed before
extending patch testing.
**** TODO =i586-gnu=
Builds are now happening on bordeaux.guix.gnu.org, once it's caught up,
enabling i586-gnu should be possible. There might be issues with a lack of
build resources (like x86_64-linux).
**** TODO Test reproducibility
This depends on getting more data on builds back to the data service.
Submit multiple builds for the same derivation.
*** DONE Improve patches page
**** DONE Add more statuses
**** DONE Support filtering by status
*** TODO Run licensecheck on package sources
And highlight changes or when this doesn't match what's declared in the Guix
package.
** QA Ecosystem
*** data.qa.guix.gnu.org
**** TODO Move away from Chris renting beid (the machine it runs on)
**** TODO Make processing revisions faster and require less resources
**** TODO Support accepting build reports/buildinfo
This can include the hashes of the output and be factored in to the assessment
of reproducibility. This will allow using the results of multiple builds per
build server, rather than just being able to look at a single substitute.
*** Monitoring and observability
**** TODO Move away from Chris hosting Prometheus/Grafana for this purpose
Either to Prometheus packaged for Guix and run on Guix machines, or some other
approach.
*** Keeping Guix packages up to date
**** TODO Automated patch submission for package updates
Using guix refresh. This way the manual work is reduced, and it's easier to
just apply the package updates that have been tested.
***** TODO Speed up QA for patches
As this would result in more patch series, QA (mostly the data service) will
need speeding up a lot to make this feasible.
*** Facilitating people without commit access reviewing patches
This might help speed up getting patches merged, and get more people involved
in the process. Maybe the QA Frontpage has a part to play in helping this to
happen.
**** DONE Add mark patches as "reviewed-looks-good" feature
** Reproducible builds
*** TODO Improve issue display, show severity and separate by status
*** TODO Record and expose percentage of packages that build reproducibly
Compute the percentage, ignoring packages where the status is unknown. Also
ignore systems with little data available.
Expose in Prometheus metrics, and record in the database. This could be
backfilled by looking through the older data service data.
|