Test Files ========== All tests are shell scripts and must be named to match (shell glob): t[0-9][0-9][0-9][0-9]-*.sh They must output TAP (Test Anything Protocol) compliant format. In particular, only version 12 format is supported, NOT version 13. See `perldoc Test::Harness::TAP` for extensive details on the format. A more formal grammar can be found in `perldoc TAP::Parser::Grammar` if desired. See also `perldoc TAP::Parser` and <https://testanything.org> as well for copies of the specification. All tests are run with the current directory set to the same directory as the directory containing the test file itself. ------------------------- Test Naming and Numbering ------------------------- All the test scripts are named like so: tNNNN-command-more-info.sh Where `NNNN` is a 4-digit, 0-padded decimal number. The first (i.e. leftmost) digit should indicate the category of the test as follows: 0. the testing framework itself and universal stuff 1. basic fundamental commands and options (e.g. help, status, --top-bases) 2. creation and deletion of tg branches and dependencies and hooks 3. graph navigation commands (e.g. prev, next, checkout) 4. informational and introspection commands (e.g. summary, info, base) 5. update command 6. tag and revert 7. remote and push 8. import and export 9. shortcut and utility commands (e.g. files, log, mail, patch, rebase) The second digit should indicate the command within that group. In other words if the second digit is "3" then all tests numbered 13NN should be testing the same command. Generally a given command should have all its tests in the same family. The third digit should be used for grouping tests of the same or related options of a command when the command supports a lot of options or may instead indicate a command "mode" that's being tested (e.g. `tg tag` has several different command modes). ------------ Test Library ------------ A testing library is available to facilitate writing tests. It should be sourced by test scripts wanting to use it AFTER setting the "test_description" variable and then calling the provided functions to produce TAP output like so: test_description='title of test goes here And any more lines go here much like a standard Git checkin comment although there is no requirement that the description follow any particular layout. It is only used by the -h|--help functionality. ' . ./test-lib.sh test_expect_success 'small test' ' # do some testing ... ' ... test_done For more detailed information on how to use the test-lib.sh testing library see the README-TESTLIB and README-WRITING-TESTS files. ---------------------- TAP - A Quick Overview ---------------------- Only output to STDOUT is examined and must consist of one of four kinds of lines: 1) A test plan line matching (perl regular expressions) either: a) `^1\.\.$count$` where `$count` is a positive integer b) `^1\.\.0(?![0-9]).*?#\s*(?i)SKIP\S*\s+(.*)$` where $1 is skip reason (this format is only valid if there are no test lines) There MUST BE EXACTLY ONE test plan line and it must appear either BEFORE ALL or AFTER ALL of the test lines. For example, the following line plans four tests: 1..4 2) Test lines which must either be ALL BEFORE or ALL AFTER the test plan line: a) `^ok(?:\s+$stuff?)?$` test succeeds b) `^not ok(?:\s+$stuff?)?$` test fails There must be n test lines where n (possibly 0) is from the test plan. If present, `$stuff` should match: (\d+)?\s*([^#]*)(?:#\s*(?i)(SKIP|TODO)\b(.*))? where $1 is the test number and if present must be correct (tests are numbered starting with 1). $2 is the optional test description and it's customary to start it with "- " to make the output look nice. If present, $3 is a directive and $4 is the reason for it. An "ok #TODO" is a known breakage that isn't actually broken. A "not ok #TODO" is a known breakage (that's still broken). An "ok #SKIP" is a skipped test. A "not ok #SKIP" is treated the same as a "not ok" test. For example, the following shows four test lines for good, skip, bad and known broken respectively: ok 1 - test that works ok 2 - test might work # SKIP need missing thingamajig to run not ok 3 - test that should have worked not ok 4 - test known not to work # TODO fix this problem 3) Diagnostic/Comment lines matching `^#` which are ignored for TAP purposes (If the `'#'` isn't in column 1 then it's technically an "other" line.) Some harnesses have an option to show comments and "other" lines do NOT qualify, only lines matching `^#` are considered "comments". For example, all of the following are recognized as comment/diagnostic lines: # Hello # some random gobbledygook # Lines may be located anywhere in the output 4) An emergency stop line matching `^\s*Bail out!\s*(.*)$` (yes, it IS case-sensitive). The value of $1 will be shown if present. (A well-written test emits a '1..0 # SKIP ...' line instead, but if something unrecoverable goes wrong in the middle of testing a "Bail out!" line is useful.) For example: Bail out! my microphone is broken When using prove to run multiple tests a `'Bail out!'` line will abort all further testing when it's encountered (including any yet-to-be-run tests). The handling of other lines is unspecified although generally they are treated as lines to be ignored, but should the TAP standard change there is no guarantee they will continue to be so treated.