test (e2e) : move manpage checks to a separate scenario (#4608) #4611
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
rohanKanojia
force-pushed
the
pr/fix-e2e-man
branch
from
29d7534 to
b08957d
Compare
rohanKanojia
changed the title
rohanKanojia
force-pushed
the
pr/fix-e2e-man
branch
from
b08957d to
dbc4d72
Compare
…rc-org#4608) + Update manpages step in basic scenario to skip execution for windows + Instead of relying of man command output, only verify whether we've generated the man pages files correctly in the directory. Signed-off-by: Rohan Kumar <[email protected]>
rohanKanojia
force-pushed
the
pr/fix-e2e-man
branch
from
dbc4d72 to
df2b631
Compare
test/e2e/testsuite/testsuite.go
Outdated
| manPageDir := filepath.Join(userHomeDir, ".local", "share", "man", "man1") | ||
| for _, manPage := range expectedManFileList { | ||
| manFile := filepath.Join(manPageDir, manPage) | ||
| manCommand := fmt.Sprintf("man -P cat %s", manFile) |
There was a problem hiding this comment.
should we be checking man -P cat crc instead of using the manpage filepath?
i think the generation of the files is already covered by the unit tests and here we can check for if crc setup also updated the required env variables for having the manpages directly accessible by man
There was a problem hiding this comment.
I added this instead of directly checking the command name because man wasn't discovering the ~/.local/share/man/man1 directory on the QE lab instance.
There was also a problem with MacOS as there was a package named crc that had a manpage (see #4608 (comment))
ture:16
Then executing "man -P cat crc" fails # features/manpages.feature:19
Error: command 'man -P cat crc', expected to fail, exited with exit code: 0
Command stdout: crc(n) crc(n)
______________________________________________________________________________
NAME
crc - Message digest "crc"
SYNOPSIS
package require Tcl ?8.2?
package require Trf ?2.1.4?
crc ?options...? ?data?
______________________________________________________________________________
DESCRIPTION
The command crc is one of several message digests provided by the
package trf. See trf-intro for an overview of the whole package.
crc ?options...? ?data?
The options listed below are understood by the digest if and
only if the digest is attached to a channel. See section
IMMEDIATE versus ATTACHED for an explanation of the term
attached.
-mode absorb|write|transparent
This option has to be present. The specified argument
determines the behaviour of the digest in attached mode.
Beyond the argument values listed above all unique
abbreviations are recognized too. Their meaning is
explained below:
absorb All data written to the channel is used to
calculate the value of the message digest and then
passed unchanged to the next level in the stack of
transformations for the channel the digest is
attached to. When the channel is closed the
completed digest is written out too, essentially
attaching the vlaue of the diggest after the
information actually written to the channel.
When reading from the channel a value for the
digest is computed too, and after closing of the
channel compared to the digest which was attached,
i.e. came behind the actual data. The option
-matchflag has to be specified so that the digest
knows where to store the result of said
comparison. This result is a string and either
"ok", or "failed".
write All data read from or written to the channel the
digest is attached to is ignored and thrown away.
Only a value for the digest of the data is
computed. When the channel is closed the computed
values are stored as ordered through the options
-write-destination, -write-type, -read-
destination, and -read-type.
transparent
This mode is a mixture of both absorb and write
modes. As for absorb all data, read or written,
passes through the digest unchanged. The generated
values for the digest however are handled in the
same way as for write.
-matchflag varname
This option can be used if and only if the option "-mode
absorb" is present. In that situation the argument is the
name of a global or namespaced variable. The digest will
write the result of comparing two digest values into this
variable. The option will be ignored if the channel is
write-only, because in that case there will be no
comparison of digest values.
-write-type variable|channel
This option can be used for digests in mode write or
transparent. Beyond the values listed above all their
unique abbreviations are also allowed as argument values.
The option determines the type of the argument to option
-write-destination. It defaults to variable.
-read-type variable|channel
Like option -write-type, but for option -read-
destination.
-write-destination data
This option can be used for digests in mode write or
transparent. The value data is either the name of a
global (or namespaced) variable or the handle of a
writable channel, dependent on the value of option
-write-type. The message digest computed for data written
to the attached channel is written into it after the
attached channel was closed. The option is ignored if
the channel is read-only.
Note that using a variable may yield incorrect results
under tcl 7.6, due to embedded \0's.
-read-destination data
This option can be used for digests in mode write or
transparent. The value data is either the name of a
global (or namespaced) variable or the handle of a
writable channel, dependent on the value of option -read-
type. The message digest computed for data read from the
attached channel is written into it after the attached
channel was closed. The option is ignored if the channel
is write-only.
Note that using a variable may yield incorrect results
under tcl 7.6, due to embedded \0's.
The options listed below are always understood by the digest, attached
versus immediate does not matter. See section IMMEDIATE versus ATTACHED
for explanations of these two terms.
-attach channel
The presence/absence of this option determines the main
operation mode of the transformation.
If present the transformation will be stacked onto the
channel whose handle was given to the option and run in
attached mode. More about this in section IMMEDIATE
versus ATTACHED.
If the option is absent the transformation is used in
immediate mode and the options listed below are
recognized. More about this in section IMMEDIATE versus
ATTACHED.
-in channel
This options is legal if and only if the transformation
is used in immediate mode. It provides the handle of the
channel the data to transform has to be read from.
If the transformation is in immediate mode and this
option is absent the data to transform is expected as the
last argument to the transformation.
-out channel
This options is legal if and only if the transformation
is used in immediate mode. It provides the handle of the
channel the generated transformation result is written
to.
If the transformation is in immediate mode and this
option is absent the generated data is returned as the
result of the command itself.
NOTES
This command uses the same CRC polynomial as the CRC algorithm used by
PGP (http://www.pgp.com).
IMMEDIATE VERSUS ATTACHED
The transformation distinguishes between two main ways of using it.
These are the immediate and attached operation modes.
For the attached mode the option -attach is used to associate the
transformation with an existing channel. During the execution of the
command no transformation is performed, instead the channel is changed
in such a way, that from then on all data written to or read from it
passes through the transformation and is modified by it according to
the definition above. This attachment can be revoked by executing the
command unstack for the chosen channel. This is the only way to do this
at the Tcl level.
In the second mode, which can be detected by the absence of option
-attach, the transformation immediately takes data from either its
commandline or a channel, transforms it, and returns the result either
as result of the command, or writes it into a channel. The mode is
named after the immediate nature of its execution.
Where the data is taken from, and delivered to, is governed by the
presence and absence of the options -in and -out. It should be noted
that this ability to immediately read from and/or write to a channel is
an historic artifact which was introduced at the beginning of Trf's
life when Tcl version 7.6 was current as this and earlier versions have
trouble to deal with \0 characters embedded into either input or
output.
SEE ALSO
adler, crc, crc-zlib, haval, md2, md5, md5_otp, ripemd-128, ripemd-160,
sha, sha1, sha1_otp, trf-intro
KEYWORDS
authentication, crc, hash, hashing, mac, message digest, pgp
COPYRIGHT
Copyright (c) 1996-2003, Andreas Kupries <[email protected]>
Trf transformer commands 2.1.4 crc(n)
There was a problem hiding this comment.
additionally providing the manpage section number seems to access the correct crc manpage,
this works: man -P cat 1 crc
I added this instead of directly checking the command name because man wasn't discovering the ~/.local/share/man/man1 directory on the QE lab instance.
this seems to be working locally on fedora, so maybe we can investigate and fix it also for the QE lab machine
There was a problem hiding this comment.
okay, I've updated it as per your suggestion.
test/e2e/testsuite/testsuite.go
Outdated
| "crc-config-get.1.gz", "crc-ip.1.gz", "crc-version.1.gz", | ||
| "crc-config-set.1.gz", "crc-oc-env.1.gz", "crc.1.gz", | ||
| "crc-config-unset.1.gz", "crc-podman-env.1.gz", | ||
| "crc-config-view.1.gz", "crc-setup.1.gz", |
There was a problem hiding this comment.
and we could also use Scenario examples to hold this list of manpages, that way its part of the feature file and easy to add new ones, e.g:
crc/test/e2e/features/config.feature
Lines 9 to 24 in 457b653
There was a problem hiding this comment.
I was trying this approach, but it has slower execution time as crc setup is being executed for every scenario execution.
Do you know if it's possible to make it execute only once?
@story_manpages
Feature: Check generation and cleanup of manpages
Background:
Given executing single crc setup command succeeds
@linux @darwin
Scenario Outline: verify man pages are accessible after setup
Then executing "man -P cat <crc-subcommand>" succeeds
When executing crc cleanup command succeeds
Then executing "man -P cat <crc-subcommand>" fails
@linux @darwin
Examples: Man pages to check
| crc-subcommand |
| crc-bundle-generate |
| crc-config |
| crc-start |
| crc-bundle |
| crc-console |
| crc-status |
| crc-cleanup |
| crc-delete |
| crc-stop |
| crc-config-get |
| crc-ip |
| crc-version |
| crc-config-set |
| crc-oc-env |
| crc |
| crc-config-unset |
| crc-podman-env |
| crc-config-view |
| crc-setup |
There was a problem hiding this comment.
We could split it into multiple scenarios, one Scenario where we run just crc setup and then a Scenario Outline with the examples, and then another Scenario for just crc cleanup and lastly a Scenario Outline to check the failure case examples.. but this feels messy, maybe @lilyLuLiu or @adrianriobo knows the right approach to do this
There was a problem hiding this comment.
Timing wise with this approach the test time locally for this features is:
19 scenarios (19 passed)
76 steps (76 passed)
4m3.008155834s
ok github.com/crc-org/crc/v2/test/e2e 244.039s
There was a problem hiding this comment.
We could split it into multiple scenarios, one Scenario where we run just crc setup and then a Scenario Outline with the examples, and then another Scenario for just crc cleanup and lastly a Scenario Outline to check the failure case examples
Thanks for suggestion, I was able to get this approach working. It executes in 15s
rohanKanojia
force-pushed
the
pr/fix-e2e-man
branch
from
6a26a73 to
ddff21e
Compare
Signed-off-by: Rohan Kumar <[email protected]>
rohanKanojia
force-pushed
the
pr/fix-e2e-man
branch
from
ddff21e to
0426c1c
Compare
|
@rohanKanojia: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Description
Fixes: #4608
Relates to: #4586
In #4586 , I had added step to check
man -P cat crcto verify whether man pages are correctly generated after executingcrc setupin basic scenariomanpathis not updated correctly;mancommand fails to list manpages for CRC. I see that there is an already existing entry for some crc named man page (see man pages didn't cleaned after crc cleanup #4608 (comment))Update
basic.featureto remove manpage checks and add a new scenariomanpages.featureto verify manpage generation and cleanup. This new scenario would only be enabled for@darwinand@linuxenvironments.Type of change
test, version modification, documentation, etc.)
Proposed changes
These changes update how we verify manpage generation:
Testing
I've run
manpages.featureon MacOS, Linux machine to verify that these changes don't break anything.Contribution Checklist