PR#83: Add TMT interop tests - rpms/gnutls

rpms / gnutls

#83 Add TMT interop tests

Merged a year ago by zfridric. Opened a year ago by not4pedro.

rpms/ not4pedro/gnutls interop-tests into rawhide

Peter Leitmann • a year ago

2f8c73c

.fmf/version

file added

		`@@ -0,0 +1,1 @@`
		`+ 1`

ci.fmf

file added

		`@@ -0,0 +1,1 @@`
		`+ resultsdb-testcase: separate`

plans/fips-smoke.fmf

file added

		`@@ -0,0 +1,4 @@`
		`+ summary: Runs FIPS library integrity checks.`
		`+ name: fips-smoke`
		`+ execute:`
		`+ script: if [[ $(GNUTLS_DEBUG_LEVEL=99 GNUTLS_FORCE_FIPS_MODE=1 certtool 2>&1 \| grep "Error") ]]; then exit 1; else exit 0; fi;`

plans/nss-2way.fmf

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ summary: Upstreamed gnutls-openssl interop-2way tests`
		`+ contact: Stanislav Zidek <szidek@redhat.com>`
		`+ discover:`
		`+ # upstreamed tests (public)`
		`+ - name: interop-nss-2way`
		`+ how: fmf`
		`+ url: https://gitlab.com/redhat-crypto/tests/interop.git`
		`+ filter: 'tag: interop-gnutls & tag: interop-nss & tag: interop-2way'`
		`+ execute:`
		`+ how: tmt`

plans/openssl-2way.fmf

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ summary: Upstreamed gnutls-openssl interop-2way tests`
		`+ contact: Stanislav Zidek <szidek@redhat.com>`
		`+ discover:`
		`+ # upstreamed tests (public)`
		`+ - name: interop-openssl-2way`
		`+ how: fmf`
		`+ url: https://gitlab.com/redhat-crypto/tests/interop.git`
		`+ filter: 'tag: interop-gnutls & tag: interop-openssl & tag: interop-2way'`
		`+ execute:`
		`+ how: tmt`

plans/short-interop-tests.fmf

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ summary: Upstreamed gnutls interop tests - short tests which do not need to run in parallel`
		`+ contact: Stanislav Zidek <szidek@redhat.com>`
		`+ discover:`
		`+ # upstreamed tests (public)`
		`+ - name: interop-gnutls-short`
		`+ how: fmf`
		`+ url: https://gitlab.com/redhat-crypto/tests/interop.git`
		`+ filter: 'tag: interop-gnutls & tag: -interop-slow'`
		`+ execute:`
		`+ how: tmt`

tests/tests.yml

file removed

-13

		`@@ -1,13 +0,0 @@`
		`- ---`
		`- - hosts: localhost`
		`- roles:`
		`- - role: standard-test-basic`
		`- tags:`
		`- - classic`
		`- tests:`
		`- - smoke-fips:`
		`- dir: .`
		`- run: if [[ $(GNUTLS_DEBUG_LEVEL=99 GNUTLS_FORCE_FIPS_MODE=1 certtool 2>&1 \| grep "Error") ]]; then exit 1; else exit 0; fi;`
zfridric commented a year ago I would rather not lose this test. The whole point is to just run certtool with GNUTLS_FORCE_FIPS_MODE=1 which forces the FIPS integrity checks to run and check whether the FIPS integrity checks succeed or not.
		`- required_packages:`
		`- - gnutls`
		`- - gnutls-utils`

not4pedro commented a year ago

Aiming to run interoperability tests.

rebased onto ade0783d94dd331c684a693f5a66ccb1f3579855

a year ago

rebased onto ba4245c1b92835b648a76919a02fb624e5944417

a year ago

not4pedro commented a year ago

Hi Daiki @ueno , we are working on adding TMT tests here. But there seems to be a conflict with the STI tests. The TMT tests run only when we remove the tests/tests.yml file. Are the STI tests needed? Can these tests be rewritten to TMT? thanks

rebased onto a1fa129023e1935320e94db1ceb4305d60d8a07e

a year ago

szidek commented a year ago

LGTM

szidek commented on line 9 of plans/short-interop-tests.fmf a year ago

Considering other plans, should it be enough to specify: tag: interop-gnutls & tag: -interop-2way?

Edited a year ago by szidek

not4pedro commented a year ago

Considering other plans, should it be enough to specify: tag: interop-gnutls & tag: -interop-2way?

@szidek Correct. Future-proofing?

szidek commented a year ago

Well, it's not clear what is the meaning of the interop-nss-slow in this context. I'd prefer to have the filtering rules as simple as possible.

zfridric commented on line 6 of plans/nss-2way.fmf a year ago

I think the name is incorrect. Shouldn't it be interop-nss-2way?

zfridric commented on line 11 of tests/tests.yml a year ago

I would rather not lose this test. The whole point is to just run certtool with GNUTLS_FORCE_FIPS_MODE=1 which forces the FIPS integrity checks to run and check whether the FIPS integrity checks succeed or not.

rebased onto 26982e74f9e79ba2e1c5fefa2703c4f4cc406eba

a year ago

rebased onto 285a59fd63f6977e9f86ead977d2a3e82c95a686

a year ago

rebased onto b42737c18accd44dfa6ee9cb5094e1ea7ffc90c1

a year ago

rebased onto bb947abfa649a4de94c1f8cb704e7477156f489a

a year ago

zfridric commented on line 2 of plans/fips-smoke.fmf a year ago

I think as a summary it would be enough to say "Runs FIPS library integrity checks" rather then explaining the logic of the test itself.

rebased onto bb709f9971cc91a2c175199183d246898c95eb7f

a year ago

rebased onto fdcebc85e737427b81cd66d1660336ef8a05915e

a year ago

szidek commented on line 9 of plans/nss-2way.fmf a year ago

I feel here we are using interop-parallel-1 unnecessarily, why not to replace it with interop-2way?

szidek commented on line 9 of plans/openssl-2way.fmf a year ago

Same here, could we use interop-2way instead of interop-parallel-1?

not4pedro commented a year ago

I feel here we are using interop-parallel-1 unnecessarily, why not to replace it with interop-2way?

Don't you think that this could become a mess when you add more 2way tests? With this explicit tags, the parallel tests are fully under control.

szidek commented a year ago

I feel here we are using interop-parallel-1 unnecessarily, why not to replace it with interop-2way?

Don't you think that this could become a mess when you add more 2way tests? With this explicit tags, the parallel tests are fully under control.

Not really, 2way scenarios might either be added to the same test, or there will be a complete new type in far future (e.g., TLS 1.4) and we will add separate tag for that.

I'd like to keep *-parallel-* tags only for situations where it is not reasonably possible to discern two tests with "realistic tags" (meaning tags that describe real properties of the test).

rebased onto ca85fa46d34d4e25a22c82b6e777ce1446e84a1f

a year ago

rebased onto 2f8c73c

a year ago

szidek commented a year ago

Looks good to me.

@zfridric could you please merge this?

zfridric commented a year ago

looks good