Good day,
A long time ago I (kinda) promised to do something good on the testing support for tdelibs with cmake. I'm sorry I protracted it so much. But as it sad it's better late then never, so I want to present a series of patches to fix/add/restore/enhance/whatever test support in tdelibs.
Long story short: the repository is located here: https://github.com/Fat-Zer/tdelibs/tree/fix-check (branch fix-check) Note that cmake submodule has a different link due to it required some modifications too
There were already some workarounds for tdeui and tdeabc, but they had several problems: - checks executables were added to "all" target, so they were build unconditionally - tests were run during build phase (which is confusing and wrong) - no test statistics etc
What the patch set features: - Add EXCLUDE_FROM_ALL flag for tde_add_library macro - New macro tde_add_check_executable - add tests from dcop/tdecore/tdeui/tdeio/tdeabc/tdewallet - a fix in tdeio against mimemagic (fixes one check application) - 40+ automated tests (mostly of base features which won't likely fail) but anyway IMHO it's nice to have them.
How to use: To build tests/check run "make check"; To run automated tests run either "make test" or "ctest"; To run specific tests e.g. for tdecore use "ctest -R tdecore" (note that "tdecore" is just a substring of a test name ) To see verbose output of tests add -V flag "ctest -V -R tdecore". For more information see cmake documentation about ctest and cmake add_test macro.
No automated test require nor X session nor a running tde session, but if run inside a such session they may interact with it.
About tde_add_check_executable cmake macro: The macro is mostly a tde_add_executable with a redused set of arguments, but except of adding executable target to the "all" it adds them to a special custom target "check". As of specific of test executables (which are mostly one-source-file-based) it doesn't require the SOURCE section, instead of which it globs the files based on the target name. Also it has a TEST argument to automatically add a test out of given file.
Fat-Zer wrote:
Good day,
A long time ago I (kinda) promised to do something good on the testing support for tdelibs with cmake. I'm sorry I protracted it so much. But as it sad it's better late then never, so I want to present a series of patches to fix/add/restore/enhance/whatever test support in tdelibs.
Long story short: the repository is located here: https://github.com/Fat-Zer/tdelibs/tree/fix-check (branch fix-check) Note that cmake submodule has a different link due to it required some modifications too
There were already some workarounds for tdeui and tdeabc, but they had several problems:
- checks executables were added to "all" target, so they were build
unconditionally
- tests were run during build phase (which is confusing and wrong)
- no test statistics etc
What the patch set features:
- Add EXCLUDE_FROM_ALL flag for tde_add_library macro
- New macro tde_add_check_executable
- add tests from dcop/tdecore/tdeui/tdeio/tdeabc/tdewallet
- a fix in tdeio against mimemagic (fixes one check application)
- 40+ automated tests (mostly of base features which won't likely
fail) but anyway IMHO it's nice to have them.
How to use: To build tests/check run "make check"; To run automated tests run either "make test" or "ctest"; To run specific tests e.g. for tdecore use "ctest -R tdecore" (note that "tdecore" is just a substring of a test name ) To see verbose output of tests add -V flag "ctest -V -R tdecore". For more information see cmake documentation about ctest and cmake add_test macro.
No automated test require nor X session nor a running tde session, but if run inside a such session they may interact with it.
About tde_add_check_executable cmake macro: The macro is mostly a tde_add_executable with a redused set of arguments, but except of adding executable target to the "all" it adds them to a special custom target "check". As of specific of test executables (which are mostly one-source-file-based) it doesn't require the SOURCE section, instead of which it globs the files based on the target name. Also it has a TEST argument to automatically add a test out of given file.
Wow - this sounds pretty cool. Thanks! I'll look forward to check out next.
On Wednesday 21 of December 2016 00:19:58 Fat-Zer wrote:
Good day,
A long time ago I (kinda) promised to do something good on the testing support for tdelibs with cmake. I'm sorry I protracted it so much. But as it sad it's better late then never, so I want to present a series of patches to fix/add/restore/enhance/whatever test support in tdelibs.
Long story short: the repository is located here: https://github.com/Fat-Zer/tdelibs/tree/fix-check (branch fix-check) Note that cmake submodule has a different link due to it required some modifications too
There were already some workarounds for tdeui and tdeabc, but they had several problems:
- checks executables were added to "all" target, so they were build
unconditionally
- tests were run during build phase (which is confusing and wrong)
- no test statistics etc
What the patch set features:
- Add EXCLUDE_FROM_ALL flag for tde_add_library macro
- New macro tde_add_check_executable
- add tests from dcop/tdecore/tdeui/tdeio/tdeabc/tdewallet
- a fix in tdeio against mimemagic (fixes one check application)
- 40+ automated tests (mostly of base features which won't likely
fail) but anyway IMHO it's nice to have them.
How to use: To build tests/check run "make check"; To run automated tests run either "make test" or "ctest"; To run specific tests e.g. for tdecore use "ctest -R tdecore" (note that "tdecore" is just a substring of a test name ) To see verbose output of tests add -V flag "ctest -V -R tdecore". For more information see cmake documentation about ctest and cmake add_test macro.
No automated test require nor X session nor a running tde session, but if run inside a such session they may interact with it.
About tde_add_check_executable cmake macro: The macro is mostly a tde_add_executable with a redused set of arguments, but except of adding executable target to the "all" it adds them to a special custom target "check". As of specific of test executables (which are mostly one-source-file-based) it doesn't require the SOURCE section, instead of which it globs the files based on the target name. Also it has a TEST argument to automatically add a test out of given file.
In short: great, good work!
Long: I have some problems with tests in tdelibs.
1) On Debian/Ubuntu is used WITH_GCC_VISIBILITY and tests tdewallet/backend/testbf and testsha causes FTBFS - see patch bp0016-fix-tdewallet-tests-with-hidden-visibility.diff
2) In dcop/tests/run-tests.sh is used "source ./shell.generated..." where "source" is a bash specific => test may fail with other shells (dash on Debian/Ubuntu). Fixed in bp0017-fix-dcop-test-bashism.diff
3) Test tdeio/kmimetypetest expects mimetype definitions in tdesycoca cache => fails if it is done prior to installation. Proposed patch bp0018-fix-tdeio-kmimetype-test.diff contains a workaround for the test during the compilation on a clean build environment.
4) Test tdeabc/vcardparser/testread2 fail, probably because of some erroneous utf8 conversion. For now I've had not examined it in detail == patch not included.
5) Test tdeabc/testldapclient expects dcop server, tdelauncher and access to a foreign ldap server on the Internet => fails if it is done prior to installation and depends on many factors which can not be influenced. Therefore, I suggest not to perform this test during compilation - see patch bp0019-fix-tdeabc-ldapclient-test.diff
Cheers
On Sunday 22 of January 2017 03:16:05 Slávek Banko wrote:
- Test tdeabc/vcardparser/testread2 fail, probably because of some
erroneous utf8 conversion. For now I've had not examined it in detail == patch not included.
Patch for testread2 attached.
2017-01-22 5:16 GMT+03:00 Slávek Banko slavek.banko@axis.cz:
On Wednesday 21 of December 2016 00:19:58 Fat-Zer wrote: In short: great, good work!
Long: I have some problems with tests in tdelibs.
- On Debian/Ubuntu is used WITH_GCC_VISIBILITY and tests
tdewallet/backend/testbf and testsha causes FTBFS - see patch bp0016-fix-tdewallet-tests-with-hidden-visibility.diff
- In dcop/tests/run-tests.sh is used "source ./shell.generated..."
where "source" is a bash specific => test may fail with other shells (dash on Debian/Ubuntu). Fixed in bp0017-fix-dcop-test-bashism.diff
- Test tdeio/kmimetypetest expects mimetype definitions in tdesycoca
cache => fails if it is done prior to installation. Proposed patch bp0018-fix-tdeio-kmimetype-test.diff contains a workaround for the test during the compilation on a clean build environment.
- Test tdeabc/vcardparser/testread2 fail, probably because of some
erroneous utf8 conversion. For now I've had not examined it in detail == patch not included.
- Test tdeabc/testldapclient expects dcop server, tdelauncher and access
to a foreign ldap server on the Internet => fails if it is done prior to installation and depends on many factors which can not be influenced. Therefore, I suggest not to perform this test during compilation - see patch bp0019-fix-tdeabc-ldapclient-test.diff
Slavek, thanks for reviewing and debugging all the errors.
I agree with and approve 4 patches out of 5: all except bp0018-fix-tdeio-kmimetype-test.diff.
IMO the script is potentially error-prone, e.g. it may slightly break some-ones installation if .trinity/share/mimelnk is already a symlink. And in general it's quite kludgy workround... All in all I would prefer just to disable the test instead of fixing it that way...
Should I push the patches to the repository? or are you going to merge the patches soon?
Dne út 24. ledna 2017 Fat-Zer napsal(a):
2017-01-22 5:16 GMT+03:00 Slávek Banko slavek.banko@axis.cz:
On Wednesday 21 of December 2016 00:19:58 Fat-Zer wrote: In short: great, good work!
Long: I have some problems with tests in tdelibs.
- On Debian/Ubuntu is used WITH_GCC_VISIBILITY and tests
tdewallet/backend/testbf and testsha causes FTBFS - see patch bp0016-fix-tdewallet-tests-with-hidden-visibility.diff
- In dcop/tests/run-tests.sh is used "source ./shell.generated..."
where "source" is a bash specific => test may fail with other shells (dash on Debian/Ubuntu). Fixed in bp0017-fix-dcop-test-bashism.diff
- Test tdeio/kmimetypetest expects mimetype definitions in tdesycoca
cache => fails if it is done prior to installation. Proposed patch bp0018-fix-tdeio-kmimetype-test.diff contains a workaround for the test during the compilation on a clean build environment.
- Test tdeabc/vcardparser/testread2 fail, probably because of some
erroneous utf8 conversion. For now I've had not examined it in detail == patch not included.
- Test tdeabc/testldapclient expects dcop server, tdelauncher and
access to a foreign ldap server on the Internet => fails if it is done prior to installation and depends on many factors which can not be influenced. Therefore, I suggest not to perform this test during compilation - see patch bp0019-fix-tdeabc-ldapclient-test.diff
Slavek, thanks for reviewing and debugging all the errors.
I agree with and approve 4 patches out of 5: all except bp0018-fix-tdeio-kmimetype-test.diff.
IMO the script is potentially error-prone, e.g. it may slightly break some-ones installation if .trinity/share/mimelnk is already a symlink. And in general it's quite kludgy workround... All in all I would prefer just to disable the test instead of fixing it that way...
Should I push the patches to the repository? or are you going to merge the patches soon?
I updated patch bp0018-fix-tdeio-kmimetype-test.diff so that the test is performed with a specific profile. It is acceptable?
I suppose I will push patches soon.
Cheers
2017-01-24 19:32 GMT+03:00 Slávek Banko slavek.banko@axis.cz:
Dne út 24. ledna 2017 Fat-Zer napsal(a):
I updated patch bp0018-fix-tdeio-kmimetype-test.diff so that the test is performed with a specific profile. It is acceptable?
I suppose I will push patches soon.
Cheers
Slávek
ok, can't say anything against this one =) looks much better...
On Wednesday 01 of February 2017 14:57:28 Fat-Zer wrote:
2017-01-24 19:32 GMT+03:00 Slávek Banko slavek.banko@axis.cz:
Dne út 24. ledna 2017 Fat-Zer napsal(a):
I updated patch bp0018-fix-tdeio-kmimetype-test.diff so that the test is performed with a specific profile. It is acceptable?
I suppose I will push patches soon.
Cheers
Slávek
ok, can't say anything against this one =) looks much better...
In a recent commit I performed another modification: tests that use DCOP are now performed with a temporary home. It solves the problem for builds in chroot environment with read-only home directory - as is for example sbuild and pbuilder >= 0.220.
Thank you very much for your good work on tests. Thanks to tests showed that on some places is missing detection whether the system is big-endian and the code may be built wrongly. Affected are primarily architectures MIPS and PowerPC.
Cheers
-
deloptes -
Fat-Zer -
Slávek Banko