Skip to content

Add support for writing the tests output in JSON format #6010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+57 −1
Open

Add support for writing the tests output in JSON format #6010

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
19e61c2
Add support for writing the tests output in JSON format
NSCoder Jan 1, 2023
b9785bb
Update implementation to use codable
NSCoder Jan 4, 2023
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 57 additions & 1 deletion Sources/Commands/SwiftTestTool.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import CoreCommands
import Dispatch
import class Foundation.NSLock
import class Foundation.ProcessInfo
import class Foundation.JSONEncoder
import PackageGraph
import PackageModel
import SPMBuildCore
Expand Down Expand Up @@ -111,6 +112,10 @@ struct TestToolOptions: ParsableArguments {
help: "Path where the xUnit xml file should be generated.")
var xUnitOutput: AbsolutePath?

@Option(name: .customLong("json-output"),
help: "Path where the json output file should be generated.")
var jsonOutput: AbsolutePath?
Copy link
Contributor

@tomerd tomerd Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this flag should be mutually exclusive with with xUnitOutput?
Copy link
Contributor

@MaxDesiatov MaxDesiatov Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would a better alternative be to introduce a single option through which users can specify test output format instead of introducing flags for every single format?
Copy link
Author

@NSCoder NSCoder Jan 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good questions, I don't see a reason why they have to be mutually exclusive besides modifying the options from the command to only allow one output.

@MaxDesiatov I was thinking about this as well, but giving that I'm not familiar with the project I decided to keep this patch it as simple as possible. Would it be all right if we try to add the output option refactor in a following up patch?
Copy link
Contributor

@tomerd tomerd Jan 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I think it is fine for now

/// Generate LinuxMain entries and exit.
@Flag(name: .customLong("testable-imports"), inversion: .prefixedEnableDisable, help: "Enable or disable testable imports. Enabled by default.")
var enableTestableImports: Bool = true
Expand Down Expand Up @@ -307,6 +312,15 @@ public struct SwiftTestTool: SwiftCommand {
try generator.generate(at: xUnitOutput)
}

// Generate json file if requested
if let jsonOutput = options.jsonOutput {
let generator = JSONGenerator(
fileSystem: swiftTool.fileSystem,
results: testResults
)
try generator.generate(at: jsonOutput)
}

// process code Coverage if request
if self.options.enableCodeCoverage {
try processCodeCoverage(testProducts, swiftTool: swiftTool)
Expand Down Expand Up @@ -578,6 +592,8 @@ struct UnitTest {
}
}

extension UnitTest: Encodable {}
Copy link
Contributor

@tomerd tomerd Jan 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we go for an XUnit like format, we should probably create utility structs which will be the ones that are encoded. there will be a small cost of crating a second representation in memory, but I think it is worth it since it will give us a degree of freedom between the "real" model and the one we want to write to disk as JSON. its a technique I often use when build server JSON APIs so I know it works well.
Copy link
Contributor

@tomerd tomerd Jan 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, we can then also modify the XUnit generator to use these XUnit specific representations which will make that consistent across the two on-disk representations

/// A class to run tests on a XCTest binary.
///
/// Note: Executes the XCTest with inherited environment as it is convenient to pass senstive
Expand Down Expand Up @@ -690,6 +706,24 @@ final class TestRunner {
}
}

extension DispatchTimeInterval: Encodable {
/// Encodable conformance
/// - Note: `DispatchTimeInterval` will be encoded in nanoseconds if it does have a value or nil.
/// - Parameter encoder: An encoder instance
public func encode(to encoder: Encoder) throws {
var container = encoder.singleValueContainer()

/// Encode the time interval in nanoseconds
guard let nanoseconds = nanoseconds() else {
return try container.encodeNil()
}

try container.encode(nanoseconds)
}
}

extension ParallelTestRunner.TestResult: Encodable {}

/// A class to run tests in parallel.
final class ParallelTestRunner {
/// An enum representing result of a unit test execution.
Expand Down Expand Up @@ -1000,11 +1034,33 @@ fileprivate extension Array where Element == UnitTest {
}
}

// JSON file generator for a swift-test run.
final class JSONGenerator {
typealias TestResult = ParallelTestRunner.TestResult

/// The file system to use.
let fileSystem: FileSystem

/// The test results.
let results: [TestResult]

init(fileSystem: FileSystem, results: [TestResult]) {
self.fileSystem = fileSystem
self.results = results
}

/// Encode the test results in `.JSON` format and write them to the provided path
func generate(at path: AbsolutePath) throws {
let data = try JSONEncoder().encode(results)
try self.fileSystem.writeFileContents(path, data: data)
}
}

/// xUnit XML file generator for a swift-test run.
final class XUnitGenerator {
typealias TestResult = ParallelTestRunner.TestResult

/// The file system to use
/// The file system to use.
let fileSystem: FileSystem

/// The test results.
Expand Down