Return to Question

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

 

• Measure voltage over resistor. ~1Vrms ?
• In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
• U(f) Spectrum flat for f > 10 Hz?
• ...

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

• Measure voltage over resistor. ~1Vrms ?
• In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
• U(f) Spectrum flat for f > 10 Hz?
• ...

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

• Measure voltage over resistor. ~1Vrms ?
• In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
• U(f) Spectrum flat for f > 10 Hz?
• ...

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems:

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 don't need to run if Module X hasn't changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting.

I am personally not wholly sold on the "use HTML documents and track the test documents in git", but I agree with the other guys that if it works it's better than Word documents.

We are open to other solutions, but only with minimal custom implementation efforts ATM. e.g. installing dedicated software and configuring some templates would be great, but we have no resources (nor patience)for "hack something together with XML and stylesheets and a database".

Some corner stones: source control is git, 6 software developers, ~1.5KLoc, one run through full manual test suite takes 4 man-weeks (OOM), but that's rarely run.

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.