Split optional dependency-backed features into packages#1172
Split optional dependency-backed features into packages#1172PrzemyslawKlys merged 55 commits intomasterfrom
Conversation
- move Playwright and ImageSharp typosquatting support into DomainDetective.Visual - move PgpCore-backed security.txt verification into DomainDetective.Pgp - prepare the CtSql registration seam without extracting the live PostgreSQL implementation yet - add package-specific NuGet readmes and align the build scripts with the new package defaults
| You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Code ReviewThis is a well-structured refactoring. The registration-seam pattern is clean, backward-compatible, and extensible. A few items worth addressing: Bug: |
- preserve original image dimensions when building visual fingerprints - dispose the PGP verifier and tighten optional feature registration state - replace broad visual catch clauses with targeted exception handling - add Visual and CtSql to the solution for IDE and solution-build parity
PR Review: Split optional dependency-backed features into packagesOverall this is a clean architectural improvement. Moving Playwright, ImageSharp, and PgpCore out of the base package is the right call for trimming/AOT scenarios and for consumers who don't need those features. The delegate-registry pattern in Bugs / Correctness1. Playwright var context = await browser.NewContextAsync(...).ConfigureAwait(false); var page = await context.NewPageAsync().ConfigureAwait(false);
await using var context = await browser.NewContextAsync(...).ConfigureAwait(false);2. Linked using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); linkedCts.CancelAfter(options.BrowserCaptureTimeout); // ... but linkedCts.Token is never passed to any Playwright callThe timeout fires but nothing observes Minor Issues3. Unnecessary return (fingerprint.Value.FingerprintHex.Trim(), fingerprint.Value.Width, fingerprint.Value.Height);
4. if (DomainDetectiveOptionalFeatures.TryVerifySecurityTxtSignature(txt, pgpPublicKey!, ...))
Design Observations5. Silent overwrite on repeated All three _securityTxtPgpVerifier = verifier ?? throw new ArgumentNullException(nameof(verifier));In production this is usually fine (startup runs once), but in test suites it can cause subtle ordering dependencies when multiple tests register different providers. Consider at least throwing 6. public static bool HasVisualProvider => _typosquattingVisualFingerprintBuilder != null && _typosquattingBrowserCapture != null;Two separate Positive ObservationsAccidental bug fix in The original
Correct change.
Clean approach for distinguishing "not provided", "true", and "false" for Questions
|
- dispose the Playwright browser context and remove the unused linked cancellation source - drop no-op fingerprint trimming in typosquatting visual analysis - remove the redundant null-forgiving operator in security.txt verification
PR Review: Split optional dependency-backed features into packagesGood structural change overall — the registration pattern is clean and the public API surface is preserved. A few items worth addressing before merge: Bug / Behavioral change: browser-capture timeout coverage regressedOld code ( using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); linkedCts.CancelAfter(options.BrowserCaptureTimeout);The linked CTS applied the timeout to the entire operation, including New code ( Suggestion: either restore the linked-CTS approach inside Good catch: image dimensions fixIn the old No guard against double-registration
public static void RegisterPgpVerifier(Func<string, string, (bool, string)> verifier) { if (_securityTxtPgpVerifier != null) throw new InvalidOperationException("A PGP verifier is already registered."); _securityTxtPgpVerifier = verifier ?? throw new ArgumentNullException(nameof(verifier)); } |
| Area | Finding | Severity |
|---|---|---|
DomainDetectiveVisualProvider | Browser launch timeout not covered | Medium |
DomainDetectiveVisualProvider | Image dimension fix (was a bug) | Positive |
DomainDetectiveOptionalFeatures | No double-registration guard | Low |
DomainDetective.Tests/app.config | Binding redirect removed — verify net472 | Low |
TyposquattingVisualSimilarity.Capture.cs | Bare catch swallows cancellation | Low |
DomainDetective.CtSql NuGet README | Scaffold nature not clearly flagged | Nit |
The registration/delegation architecture is solid and the separation of concerns is well-executed. The main item to resolve before merging is the timeout regression in the browser capture path.
- wrap Playwright startup and capture calls in the same browser timeout window - keep user-triggered cancellation propagating out of the core browser capture wrapper
PR Review: Split optional dependency-backed features into packagesOverall this is a well-structured refactor. The plugin-registration pattern is clean, the separation of concerns is clear, and the build-script improvements (structured plan output, Bug: |
- make the evidence-refresh profile self-contained - explicitly reset CT source flags alongside the other profile toggles - extend the profile test to assert the resets
- make the evidence-refresh profile enable exact-host passive CT metadata hydration by default - align the XML summary with the actual profile behavior - extend the profile test to prove the flag transition
Code ReviewOverall this is a well-structured refactor that cleanly separates optional heavy dependencies (PGP, Playwright, ImageSharp) into opt-in packages. The architecture is sound and the new tests add meaningful coverage. A few items worth discussing: Issues / Bugs1. Hardcoded IP for crt.sh PostgreSQL ( // Hostname defaults to IP 91.199.212.73The 2. The change from all fields must be missing to any field missing is semantically correct for completeness checking. However, on an existing dataset where prior runs only populated a subset of fields (e.g., thumbprint present but EKU absent), this will now trigger a backfill query for every such entry on the next run. This could cause a significant burst of crt.sh PostgreSQL queries on first run after the upgrade. Worth calling out explicitly in the PR description / migration notes, and potentially worth adding a rate-limiting safeguard or a one-time migration note. Design Concerns3.
// Pgp/Visual pattern: DomainDetectivePgpRegistration.Register(); // self-contained // CtSql pattern: DomainDetectiveCtSqlRegistration.Register(myProviderDelegate); // caller must supply implThis is intentional (the real implementation lives outside this repo), but it creates an asymmetric API surface. Consider adding a XML doc comment to 4. Broad catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { throw; } catch { return null; // swallows all exceptions silently }Swallowing all exceptions (including Code Quality5. Reflection in test is fragile
6. Minor: Using Positive Highlights
Review generated with Claude Sonnet 4.6 |
PR Review: Split optional dependency-backed features into packagesGreat architectural direction overall. The delegate-registry pattern cleanly isolates heavy dependencies while preserving full backward compatibility. A few things worth looking at before merge: Resource leak in PGP verifierFile: var keys = new EncryptionKeys(publicKey); // no using / no dispose using var pgp = new PGP(keys);
Non-atomic visual provider registrationFile:
_typosquattingVisualFingerprintBuilder = fingerprintBuilder ...; _typosquattingBrowserCapture = browserCapture ...; // second writeBetween these two stores, Unused net472 assembly references in scaffold packageFile: <ItemGroup Condition="'$(TargetFramework)' == 'net472'"> <PackageReference Include="System.Threading.Tasks.Extensions" Version="4.6.0" /> <Reference Include="System.IO.Compression" /> <Reference Include="System.IO.Compression.FileSystem" /> <Reference Include="System.ComponentModel.DataAnnotations" /> </ItemGroup>The CtSql package has no implementation yet. These look like a copy-paste from another Double timeout in browser captureFile: The method creates a linked CTS with using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); linkedCts.CancelAfter(options.BrowserCaptureTimeout); // ... await page.GotoAsync(url, new PageGotoOptions { Timeout = (float)options.BrowserCaptureTimeout.TotalMilliseconds // same value }).WaitAsync(linkedCts.Token)...Both paths are caught ( Silent overwrite on double-registration
Reflection-based tests on private methodsFile: Two new tests call Minor: IgnoreHTTPSErrors coupling to DisableTlsValidationFile: IgnoreHTTPSErrors = options.HttpRequestOptions.DisableTlsValidationIntentional for typosquatting analysis (squatted domains often have invalid certs), but a short inline comment would prevent a future reader from reading this as a mistake. What looks solid
|
- Add live eurofins.com integration coverage for exact CT metadata and passive CT subdomain discovery.\n- Request CertSpotter certificate DER for exact metadata rescue and hydrate certificate fields from it.\n- Address review feedback for atomic visual provider registration and CtSql scaffold references.
Code Review: Split Optional Dependency-Backed Features Into PackagesThis is a well-structured refactoring that cleanly isolates heavy optional dependencies into separate NuGet packages. The overall architecture is sound and the plugin pattern is a good fit for this use case. Here's detailed feedback: Architecture & Design — Positive
Issues Found1. Silent Signature Skip — Security Risk ( |
| Area | Verdict |
|---|---|
| Architecture | ✅ Clean plugin pattern, good separation of concerns |
| Backward Compatibility | ✅ Fully maintained |
| Security | |
| Thread Safety | Interlocked preferred |
| Input Validation | |
| Testability | |
| Code Style | ✅ Consistent with existing codebase patterns |
The main items worth addressing before merge are the silent PGP skip (#1) and the missing null guards (#3). The thread-safety and test isolation points (#2, #4) are lower priority but good to fix before this becomes a widely-used pattern in the codebase.
- Warn when signed security.txt content is parsed without the optional PGP verifier registered.\n- Add PGP provider input guards.\n- Cover the fallback warning path on net8.0 and net472.
Add CT ingestion planning groundwork
Code Review: Split optional dependency-backed features into packagesOverall this is a clean, well-motivated PR. Isolating Architecture & DesignStrengths:
Specific observations1. Redundant null guard in
if (signedText == null) throw new ArgumentNullException(...); if (string.IsNullOrWhiteSpace(signedText)) throw new ArgumentException(...);The 2.
// CtSql already has InternalsVisibleTo — could call: CertificateInventoryCapture.NormalizeCtMetadataCandidate(name, preserveWildcard: true)Alternatively, the method could be promoted to 3.
internal sealed class CrtShPostgreSqlMetadataProvider : ICtSqlMetadataProvider { private readonly PostgreSql _client = new();
4. <Description>Prepared package shell for future CT SQL components in DomainDetective.</Description>This description says "prepared package shell" but 5. The test 6. Connection string concatenation (low risk, worth noting)
return PostgreSql.BuildConnectionString(...) + $";Timeout={timeoutSeconds};Command Timeout={timeoutSeconds}";Appending raw key-value pairs to an opaque connection string is fragile if Positive callouts
Summary: Items 2–4 are the most actionable before merge. Items 1 and 6 are minor style/robustness nits. Item 5 (test coverage for CtSql) can reasonably follow in a subsequent PR if the package is intentionally "prepared but not yet battle-tested." No blocking issues found. |
Code Review — Split optional dependency-backed features into packagesThis is a well-motivated architectural change. Isolating heavy optional dependencies (PgpCore, Playwright, ImageSharp, Npgsql/DBAClientX) into separate NuGet packages is the right call — it keeps the base package lean and avoids forcing consumers to carry transitive dependencies they don't use. The registration-seam pattern in Static global registration state
|
Code ReviewOverall this is a well-structured PR with clear architectural intent. Separating heavy optional dependencies (PGP, Playwright/ImageSharp, PostgreSQL) into dedicated packages is the right call and the registration seam pattern in Below are observations from most to least critical. Bug / Correctness
var result = verifier(signedText, publicKey!);PgpCore will throw (e.g. Suggest wrapping the delegate call in a Design / API
Consider using distinct method names — e.g.
The public Performance / CorrectnessDouble-budget timeout in
linkedCts.CancelAfter(options.BrowserCaptureTimeout); // whole-method budget // ... browser launch, context creation consume some of that budget ... await page.GotoAsync(url, new PageGotoOptions { Timeout = (float)options.BrowserCaptureTimeout.TotalMilliseconds // same value, not remaining time })
Code QualityRedundant null check before
if (signedText == null) throw new ArgumentNullException(...); if (string.IsNullOrWhiteSpace(signedText)) throw new ArgumentException(...);
Brace style inconsistency between packages AGENTS.MD specifies K&R/OTBS for C#. The new package files ( SecurityEmpty password in default PostgreSQL connection string
PostgreSql.BuildConnectionString( host: "crt.sh", database: "certwatch", username: "guest", password: string.Empty, // guest user on public crt.sh ...)This is the documented public crt.sh read-only guest interface, so an empty password is expected and correct. The concern is that callers may copy this pattern when setting up a private mirror. A doc comment (or README note) that the guest/empty-password defaults are specific to the public crt.sh instance would reduce the risk of misconfigured private deployments. Test Coverage
Minor
Good work on the architecture overall. The core abstraction is sound and the scaffold approach for CtSql is a sensible way to avoid churn before the live implementation lands. |
Summary
DomainDetective.PgpDomainDetective.VisualDomainDetective.CtSqlpackage and registration seam without moving the live PostgreSQL implementation yetNotes
DomainDetectiveAPI stays intact; optional features are activated by registering the new packagesDomainDetective.CtSqlis intentionally scaffold-only in this PR so the active CT/PostgreSQL work can move later without churnDomainDetective.Visualsupports image fingerprinting on .NET Framework targets; browser capture requires .NET 8 or laterReal CT validation
eurofins.commetadata was validated live through CertSpotter with DER hydration into thumbprint, subject, issuer, serial, validity, authentication profile, and CT source fieldseurofins.comwas validated live through crt.sh and returned real subdomain candidatesValidation
dotnet build DomainDetective.Visual/DomainDetective.Visual.csproj -c Debug --framework net8.0dotnet build DomainDetective.Pgp/DomainDetective.Pgp.csproj -c Debug --framework net8.0dotnet build DomainDetective.CtSql/DomainDetective.CtSql.csproj -c Debug --framework net472dotnet test DomainDetective.Tests/DomainDetective.Tests.csproj -c Debug --framework net10.0 --filter "FullyQualifiedName~SignedSecurityTxtWarnsWhenPgpVerificationIsUnavailable|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_ComputesThumbprintFromDownloadedCertificate|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_AcceptsWildcardSubjectAlternativeNames"dotnet test DomainDetective.Tests/DomainDetective.Tests.csproj -c Debug --framework net8.0 --filter "FullyQualifiedName~SignedSecurityTxtWarnsWhenPgpVerificationIsUnavailable|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_ComputesThumbprintFromDownloadedCertificate|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_AcceptsWildcardSubjectAlternativeNames"dotnet test DomainDetective.Tests/DomainDetective.Tests.csproj -c Debug --framework net472 --filter "FullyQualifiedName~SignedSecurityTxtWarnsWhenPgpVerificationIsUnavailable|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_ComputesThumbprintFromDownloadedCertificate|FullyQualifiedName~TryHydrateExactCtMetadataThumbprintFromCrtShCertificateAsync_AcceptsWildcardSubjectAlternativeNames"DOMAINDETECTIVE_INTEGRATION=1 dotnet test DomainDetective.Tests/DomainDetective.Tests.csproj -c Debug --framework net8.0 --filter "FullyQualifiedName~CaptureAsync_CtEvidenceRefreshProfile_HydratesEurofinsExactHostFromLivePassiveCtSources|FullyQualifiedName~CaptureAsync_CtDiscoveryOnlyProfile_DiscoversEurofinsSubdomainsFromLivePassiveCtSources"Native CT validation
eurofins.comwith--enable-native-ct-logs --native-ct-log-only, no passive crt.sh/CertSpotter fallback, and bounded log/entry capseurofins.comsubdomains in the sampled tail window, which matches the design constraint: direct CT logs are append-only streams and need cursoring/indexing/backfill depth to become dependable domain search