c84a4c1
@@ -0,0 +1,1 @@
+ 1
@@ -1,3 +1,5 @@
/*/
/ruby-*.tar.xz
/*.rpm
+
+ !/plans/
@@ -0,0 +1,6 @@
+ summary: Test plan with all Fedora tests
+ discover:
+ how: fmf
+ url: https://src.fedoraproject.org/tests/ruby.git
+ execute:
+ how: tmt
Adding fmf plan for Fedora CI to pick up tests.
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
@isenfeld thanks for your PR to adding the test! I am happy to review.
Where can I see the log and result for the added "summary: Test plan with all Fedora tests"? Is it the "Zuul" - rpm-tmt-test? Or the "Fedora CI - dist-git test"?
https://artifacts.dev.testing-farm.io/8bf18d80-a9ce-4ed7-a19f-4971ff72c29d/
No metadata found in the '/var/ARTIFACTS/work-allbCxKWt/plans/all/discover/default/tests/' directory. Use 'tmt init' to get started.
Could you guide me how to test this PR with the tests/ruby PR on my local?
I am not knowledgeable about the CI's tmt execution on PRs, however, what you linked leads me to believe it used the main branch:
discover how: fmf url: https://src.fedoraproject.org/tests/ruby.git hash: 831c7db
and it can't find metadata in that repo, which IIRC is .fmf/version. You can simulate tmt's discover step manually by running git clone ${url}; git checkout ${hash} and then validate contents of the directory, both of the variables should be replaced with what is in the code block I provided.
.fmf/version
git clone ${url}; git checkout ${hash}
So this PR should be passing after https://src.fedoraproject.org/tests/ruby/pull-request/2# is merged which should add the required files.
This should be the correct command replicating what is in the tmt plan PR(#141 here in rpms/ruby) tmt run discover --how fmf --url="https://src.fedoraproject.org/tests/ruby.git" execute --how=tmt
tmt run discover --how fmf --url="https://src.fedoraproject.org/tests/ruby.git" execute --how=tmt
And to test it with the PR in tests/ruby, you have to run a similar command, but reference the PR fork and branch where the change is coming from:
$ tmt run discover --how fmf --url="https://src.fedoraproject.org/forks/isenfeld/tests/ruby.git" --ref main execute --how=tmt
Running only discover step:
$ tmt run discover --how fmf --url="https://src.fedoraproject.org/forks/isenfeld/tests/ruby.git" --ref main /var/tmp/tmt/run-008 /plans/default discover how: fmf url: https://src.fedoraproject.org/forks/isenfeld/tests/ruby.git ref: main warn: /run-basic-rails-application: - 'category' does not match any of the regexes: '^extra-' warn: /systemtap-static-probes-in-ruby: - 'category' does not match any of the regexes: '^extra-' warn: /environment-variable-RPM-ARCH-not-defined: - 'category' does not match any of the regexes: '^extra-' summary: 3 tests selected
Which seems like the expected results for the discover step.
I am using the following command to run it: (not sure if it is 1:1 how it would behave in CI)
$ tmt run -avvvvv discover --how fmf --url="https://src.fedoraproject.org/forks/isenfeld/tests/ruby.git" --ref main \ provision --how container -i fedora:rawhide \ execute --how tmt \ report --how display
I am specifying the provision step, to run it on Fedora Rawhide container instead of in a VM which is the default.
Using --help on anything in tmt gives some info on how to use and options, you get relevant info on tmt --help, tmt run --help, tmt run provision --help and so on... You can even add --help behind any step in the command in the last code block I wrote to get possible values.
--help
tmt
tmt --help
tmt run --help
tmt run provision --help
To have verbose output, I usually add -vvvvv behind -a after run, so I usually run tmt run -avvvvv, (the help page says, combining -v multiple times raises verbosity, not sure how much is the maximum after which it stops adding output though)
-vvvvv
-a
run
tmt run -avvvvv
-v
I'd just like to confirm, it is working for me, are these the correct steps? :)
it can't find metadata in that repo, which IIRC is .fmf/version.
As the tests/ruby repo added the .fmf/version file at the https://src.fedoraproject.org/tests/ruby/c/2af06baec0676195a9a732498c5dcc2359cd56f3?branch=main . I think we can test this PR on Fedora CI automatically or by typing recheck in a comment.
recheck
It seems that the same error is raising after adding the .fmf/version in tests/ruby. I am not sure why.
[citest]
@isenfeld Just in case, can you rebase on the lastest rawhide branch?
It seems your current source branch rawhide is Thu Sep 29 2022 Vít Ondruch <vondruch@redhat.com> - 3.1.2-170. https://src.fedoraproject.org/fork/isenfeld/rpms/ruby/blob/rawhide/f/ruby.spec#_1552 The latest rawhide is Thu Dec 08 2022 Vít Ondruch <vondruch@redhat.com> - 3.1.3-173.
Thu Sep 29 2022 Vít Ondruch <vondruch@redhat.com> - 3.1.2-170
Thu Dec 08 2022 Vít Ondruch <vondruch@redhat.com> - 3.1.3-173
rebased onto 386340c2341211aa90918ac61d0049971933b2c0
It's rebased.
Thanks for rebasing.
Possibly, the failures at Fedora CI - scratch build comes from new tzdata RPM package version that is one of the dependencies of ruby.
1) Failure: TestTimeTZ#test_gen_Asia_Singapore_45 [/builddir/build/BUILD/ruby-3.1.3/test/ruby/test_time_tz.rb:400]: TZ=Asia/Singapore Time.local(1981, 12, 31, 23, 59, 59). <"1981-12-31 23:59:59 +0730"> expected but was <"1982-01-01 00:29:59 +0800">. 2) Failure: TestTimeTZ#test_gen_Asia_Singapore_22 [/builddir/build/BUILD/ruby-3.1.3/test/ruby/test_time_tz.rb:382]: TZ=Asia/Singapore Time.utc(1981, 12, 31, 16, 29, 59).localtime. <"1981-12-31 23:59:59 +0730"> expected but was <"1982-01-01 00:29:59 +0800">. 3) Failure: TestTimeTZ#test_asia_singapore [/builddir/build/BUILD/ruby-3.1.3/test/ruby/test_time_tz.rb:143]: TZ=Asia/Singapore Time.local(1981, 12, 31, 23, 59, 59). <"1981-12-31 23:59:59 +0730"> expected but was <"1982-01-01 00:29:59 +0800">. Finished tests in 835.769179s, 25.8804 tests/s, 3292.2571 assertions/s. 21630 tests, 2751567 assertions, 3 failures, 0 errors, 66 skips ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [aarch64-linux] make: *** [uncommon.mk:830: yes-test-all] Error 3 error: Bad exit status from /var/tmp/rpm-tmp.zCI5wr (%check)
And the patch is here. Maybe. https://github.com/ruby/ruby/commit/58cc3c9f387dcf8f820b43e043b540fa06248da3
I talked with Vit yesterday. He will fix it on rpms/ruby. After that, I ask you to rebase again on the latest rawhide branch. And we see the tests/ruby result on this PR.
@isenfeld could you rebase this PR on the latest rawhide branch again? The failures above was fixed by https://src.fedoraproject.org/rpms/ruby/pull-request/144 . I expect we can see the result of tests/ruby.
rebased onto c84a4c1
Thanks for rebasing the PR. The scratch build succeeded. Now one failure on the Fedora CI is below.
https://artifacts.dev.testing-farm.io/cc706d26-7e87-4867-b621-054765addb7d/
./tmt-test-wrapper.sh: line 1: ./runtest.sh: No such file or directory Shared connection to 18.220.6.89 closed.
@isenfeld ping, what is the status here, please?
I guess that the problem is the systemtap test doesn't have fmf data in its directory, but the main.fmf is being added in this PR: https://src.fedoraproject.org/tests/ruby/pull-request/5 So perhaps once it's merged it might pass or at least should be ran by CI.
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci https://fedora.softwarefactory-project.io/zuul/buildset/75f2ba7fa7da4adca06f3972761f7a7d
@isenfeld ping, what is the status here, please? I guess that the problem is the systemtap test doesn't have fmf data in its directory, but the main.fmf is being added in this PR: https://src.fedoraproject.org/tests/ruby/pull-request/5 So perhaps once it's merged it might pass or at least should be ran by CI.
Never tried that, but Zuul enables to specify dependencies between PRs:
https://fedoraproject.org/wiki/Zuul-based-ci#How_to_specify_PR_dependencies_(runtime)
isenfeld
commented a year ago
Adding fmf plan for Fedora CI to pick up tests.