Timeline for Should test cases show desired behavior or actual/current behavior, or can they do both?
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 16, 2022 at 22:27 | comment | added | Kain0_0 | @donquixote I'm not saying you can't write it. I'm not even saying it won't work. But if I didn't know of this particular code pattern in use in the code base, my first presumption would be that the tests are flaky, and unreliable. Because when run with one flag they work, when run with the other they don't. True it is like the Progression/Regression case I highlighted, but there is a difference. Those are part of QA/Test culture, if something was titled progression I would understand that their failure would be expected, because they are being worked on now. | |
| Jan 16, 2022 at 16:20 | comment | added | Greg Burghardt | @donquixote: since your main concern is integration and functional tests, feature toggles on the tests are your best option. There is no easy answer, and you will likely need a custom solution. | |
| Jan 15, 2022 at 1:05 | comment | added | donquixote | Refined version of the above: try { self::assertDesired(); self::fail('Fixed, remove this'); } catch (AssertionFail $e) { KnownBug12345::report($e); self::assertActualBadBehavior(); }. The reference to KnownBug12345 can be used for "find usages" search. | |
| Jan 14, 2022 at 23:38 | comment | added | donquixote | Btw.. seems like a "Progression Test" becomes a "Regression Test" once the feature is implemented. | |
| Jan 14, 2022 at 21:41 | comment | added | donquixote | Tbh, I have not seen this kind of stuff anywhere else, and have not really thought it through to the end. It seems in most projects this would be a rare case anyway, because most bugs would be fixed directly. A simple // @todo comment might be good enough for the remaining cases. | |
| Jan 14, 2022 at 21:38 | comment | added | donquixote | More advanced, we could have different bugs represented as classes or methods, and then if (KnowIssue12345::exists()) { assertThis(); } else { assertThat(); }. Then a developer could look into the known issues directory and find documented bugs. When fixing a bug, they would delete the respective class and all references to it, leaving only assertThat();. Later you could look into the git history and find when exactly a documented bug was fixed. | |
| Jan 14, 2022 at 21:35 | comment | added | donquixote | "Option 3. ... Its horrible... But i could be miss reading you". In a test, I might write if (knownIssueExists()) { doThis(); } else { doThat(); }. Later, when fixing the bug, we replace the entire if/else with just ` doThat();. There could be a cli/env param to make knownIssueExists()` return FALSE - but by default, or in a CI pipeline, it would return TRUE, and perhaps write to a log somewhere to turn the result yellow (not red). | |
| Jan 14, 2022 at 8:14 | history | answered | Kain0_0 | CC BY-SA 4.0 |