PR#31: Example plan for the shared virtualenv tests - rpms/python-tox

rpms / python-tox

#31 Example plan for the shared virtualenv tests

Opened 3 years ago by psss. Modified 2 years ago

rpms/ psss/python-tox tmt into rawhide

Remove STI config in order to enable tmt tests

Petr Šplíchal • 2 years ago

1bdcdc6

Example plan for the shared virtualenv tests

Petr Šplíchal • 2 years ago

f6c72f3

.fmf/version

file added

		`@@ -0,0 +1,1 @@`
		`+ 1`

plans/venv.fmf

file added

		`@@ -0,0 +1,8 @@`
		`+ summary: Test virtual environments`
		`+ discover:`
		`+ how: fmf`
		`+ url: https://src.fedoraproject.org/forks/psss/tests/python.git`
		`+ ref: tmt`
		`+ filter: component:python-tox`
		`+ execute:`
		`+ how: tmt`

tests/tests.yml

file removed

-94

		`@@ -1,94 +0,0 @@`
		`- ---`
		`- - hosts: localhost`
		`- tags:`
		`- - classic`
		`- tasks:`
		`- - dnf:`
		`- name: "*"`
		`- state: latest`
		`-`
		`- - hosts: localhost`
		`- tags:`
		`- - classic`
		`- pre_tasks:`
		`- - import_role:`
		`- name: standard-test-source`
		`- vars:`
		`- fetch_only: True`
		`-`
		`- roles:`
		`- - role: standard-test-basic`
		`- repositories:`
		`- - repo: "https://src.fedoraproject.org/tests/python.git"`
		`- dest: "python"`
		`- - repo: "https://src.fedoraproject.org/rpms/pyproject-rpm-macros.git"`
		`- dest: "pyproject-rpm-macros"`
		`- tests:`
		`- - mock_with_tests:`
		`- dir: .`
		`- run: pyproject-rpm-macros/tests/mocktest.sh python-tox --enable-network --with tests`
		`- - all_supplementing_pythons:`
		`- dir: tests`
		`- run: ./all_supplementing_pythons.py`
		`- - smoke27:`
		`- dir: python/smoke`
		`- run: VERSION=2.7 METHOD=virtualenv ./venv.sh`
		`- - smoke35_optional:`
		`- dir: python/smoke`
		`- run: VERSION=3.5 INSTALL_OR_SKIP=true ./venv.sh`
		`- - smoke36:`
		`- dir: python/smoke`
		`- run: VERSION=3.6 ./venv.sh`
		`- - smoke37:`
		`- dir: python/smoke`
		`- run: VERSION=3.7 ./venv.sh`
		`- - smoke38:`
		`- dir: python/smoke`
		`- run: VERSION=3.8 ./venv.sh`
		`- - smoke39:`
		`- dir: python/smoke`
		`- run: VERSION=3.9 ./venv.sh`
		`- - smoke310:`
		`- dir: python/smoke`
		`- run: VERSION=3.10 ./venv.sh`
		`- - smoke311:`
		`- dir: python/smoke`
		`- run: VERSION=3.11 ./venv.sh`
		`- - smoke_pypy37:`
		`- dir: python/smoke`
		`- run: PYTHON=pypy3.7 VERSION=3.7 ./venv.sh`
		`- - smoke_pypy38:`
		`- dir: python/smoke`
		`- run: PYTHON=pypy3.8 VERSION=3.8 ./venv.sh`
		`- - smoke_pypy39:`
		`- dir: python/smoke`
		`- run: PYTHON=pypy3.9 VERSION=3.9 ./venv.sh`
		`- - pyproject_pytest:`
		`- dir: pyproject-rpm-macros/tests`
		`- run: ./mocktest.sh python-pytest`
		`- - pyproject_pluggy:`
		`- dir: pyproject-rpm-macros/tests`
		`- run: ./mocktest.sh python-pluggy`
		`- - pyproject_openqa_client:`
		`- dir: pyproject-rpm-macros/tests`
		`- run: ./mocktest.sh python-openqa_client`
		`- required_packages:`
		`- - gcc`
		`- - virtualenv`
		`- - python2.7`
		`- - python3.6`
		`- - python3.7`
		`- - python3.8`
		`- - python3.9`
		`- - python3.10-devel`
		`- - python3.11-devel`
		`- - python2-devel`
		`- - pypy3.7-devel`
		`- - pypy3.8-devel`
		`- - pypy3.9-devel`
		`- - python3-tox`
		`- - dnf`
		`- - mock`
		`- - rpmdevtools`
		`- - rpm-build`
		`- - python3-pyyaml`

psss commented 3 years ago

Demonstrates how the tmt configuration would look like.
For now fetches tests from the share tests fork.
Run tests using tmt run -v to try this out.

churchyard commented 3 years ago

One part I don't understand: Will this also run in Fedora CI (Jenkins or Zuul), or is this something I can only run manually?

psss commented 3 years ago

Yes, we want to get this integrated with Fedora CI. The Testing Farm Team is working on enabling the new pipeline which will support tmt as well. The first proof-of-concept should be ready for experimenting in the coming weeks (around July).

churchyard commented 3 years ago

So, currently, the answer is not yet. What kind of feedback would you like to see?

Should I review the PR here and ask about the code?
Should I try to run the tests locally with tmt and report back?
Should I just wait?

psss commented 3 years ago

Yes, please review the PR, try to run tests, ask questions what is not clear. Here are some hints:

Run all tests in a vm:

tmt run

Run tests in a container:

tmt run --all provision --how container

Run selected tests:

tmt run test --name regexp

Show more details:

tmt run --verbose
tmt run --debug
tmt run -vvvddd

We'll be glad for any feedback.

churchyard commented 3 years ago

Will do.

churchyard commented 3 years ago

Ehm. I seriously need to have a look at this, but it seems I kinda procrastinated this away. Sorry about that.

lbalhar commented 2 years ago

By the way, tmt is now available in Fedora CI.

churchyard commented 2 years ago

I know. We are currently experimenting with it in pipenv and might eventually switch everything. OTOH it is still not supported in Zuul and Fedora CI is unreliable alone. I'd rather switch once we have a redundancy.

churchyard commented 2 years ago

Zuul now supports this but I don't understand the change here. How do we control witch tests we run?

rebased onto 2ff4796519eac50bbf36bbeb3e23cece400f02bc

2 years ago

psss commented 2 years ago

There are several ways how to select relevant tests: Choose by test name, filter by component or tag. For the python-tox use case probably the most natural would be to mark relevant tests in the shared repo with component: python-tox and use the filter.

summary: Test virtual environments
discover:
    how: fmf
    url: https://src.fedoraproject.org/forks/psss/tests/python.git
    ref: tmt
    filter: component:python-tox
execute:
    how: tmt

I've updated both pull requests to demonstrate it. Choosing tests by name would look like this:

discover:
    how: fmf
    url: https://src.fedoraproject.org/forks/psss/tests/python.git
    ref: tmt
    test:
      - /smoke/venv/python37
      - /smoke/venv/python38
      - /smoke/venv/python39

Advantage of this approach is that the environment variable stuff and required packages for individual tests can be maintained at one place only.

1 new commit added

Remove sti

2 years ago

zuul commented 2 years ago

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging src.fedoraproject.org/rpms/python-tox for 31,21f03b91a82fc3abd9d4c88de31fccf5bb807817

rebased onto f6c72f3

2 years ago

zuul commented 2 years ago

Build succeeded.