Tests and distribution coverage

New functionality must include automated tests of its observable behavior. Add a focused regression for a bug fix; preserve compatibility cases for changed public APIs, migration identity, SQL effects, history, and failures. Follow verification for assertions, isolation, runner syntax, and evidence, and provider guidance for database semantics.

Select checks by the change

Choose the layers and distributions that can expose the change’s risks. Do not require the complete OS × architecture × runtime × database matrix for every patch.

Changed behavior or risk Required verification scope
Isolated rules, parsing, configuration Add focused Basic_tests; run the relevant surrounding suite.
Shared migration policy, history, transaction handling Add shared contracts where applicable; run SQLite and affected real providers, including failure/rollback effects. Include case-sensitive SQL Server for identifier risks.
Provider SQL, splitter, connection, bootstrap resources Test that provider with real database behavior; cover fresh setup and upgrade paths for structure changes.
CLI options, argument conversion, exit behavior Test parsing plus the affected CommandLine.* suite against a matching built executable.
Trimming, reflection, native dependencies, bundled content Exercise affected published forms and host RIDs; an ordinary library test cannot establish publish compatibility.
Public API, dependencies, packing, tool installation Validate affected packages as consumers as described below; project-reference tests alone are insufficient.
Container build, entrypoint, mounts, globalization Validate the image built from the changed source and the affected container architecture.

If required environments are unavailable, prepare the tests and report the exact unverified scope. Never replace provider evidence with a mock or call a skipped distribution check passed.

Existing test structure

Basic_tests contains isolated and CLI/configuration checks. TestCommon is shared infrastructure, not a standalone executable suite. Its Generic/ directory contains abstract behavior classes for migration folders, history, bootstrapping, and reported issues.

Concrete classes in unittests/{SqlServer,SqlServerCaseSensitive,PostgreSQL,MariaDB, Oracle,Sqlite} inherit those tests and supply provider fixtures. For example, PostgreSQL One_time_scripts inherits the shared class; provider Failing_Scripts classes override expected error prefixes/details. Check concrete overrides before claiming shared coverage.

Startup<TTestContainerDatabase,TExternalDatabase,TGrateTestContext> wires the fixture types; GrateTestContext supplies connections, dialect, SQL snippets, exception types, and capabilities. Server providers normally use Testcontainers; SQLite uses temporary local files. Guards such as SupportsTransaction and absent Sql.SleepTwoSeconds return early in shared tests. These can produce green results without exercising the named behavior; record the capability gap, not just the test count.

The five CommandLine.* and five Docker.* provider projects reuse source via CommandLine/IncludeTests.targets and Docker/IncludeTests.targets. ProjectToIncludeTestsFrom selects the provider. Imports include its migration tests, infrastructure, usings, and runner JSON; they exclude Versioning_The_Database, Failing_Scripts, and One_time_scripts. The latter two have distribution-specific classes/overrides. This does not import the complete provider suite.

Most migration SQL is generated by shared helpers using provider SQL/context. Sqlite.csproj also copies the shipped docs/skills/use-grate/assets/sqlite-example/**/* into AgentSkill/Fixtures/. AgentSkill/MigrationExamplesTests runs the built CLI assembly and checks schema, data, history, and reruns. Provider projects embed bootstrap SQL; the CLI embeds the complete consumer skill. Preserve these resource/copy rules when adding assets.

Run source and executable checks

Commands are from the repository root. Install the SDK selected by global.json, restore dependencies, and have the targeted runtimes available: Directory.Build.props targets net8.0;net9.0;net10.0. Server tests need a working Docker daemon and obtainable, host-compatible database images unless an explicitly authorized isolated external database is configured. SQLite’s ordinary tests use local files.

dotnet test --project unittests/Basic_tests --framework net10.0
dotnet test --project unittests/Sqlite --framework net10.0
dotnet test --project unittests/PostgreSQL --framework net10.0

TestCommon/Startup.cs binds environment configuration: GrateTestConfig__AdminConnectionString selects an external database; GrateTestConfig__DockerImage overrides the database image. Inspect inherited environment values before running. LogLevel controls test logging; CI uses TZ=UTC.

CommandLineGrateMigrator launches GrateExecutablePath, falling back to the assembly attribute generated in CommandLine.Common.csproj. Set an absolute path deliberately, for example:

GrateExecutablePath=/absolute/path/to/published/grate dotnet test --project unittests/CommandLine/CommandLine.PostgreSQL --framework net10.0

Use grate.exe on Windows and the shell’s environment syntax. The runner expects an executable path, not dotnet grate.dll; framework-dependent apphosts need the matching installed runtime. Build/publish the executable from the same revision as the tests, with the intended configuration, TFM, RID, and publish flags. Use the same requirement when running downloaded test DLLs with dotnet test --test-modules.

Published forms and current evidence limits

  • Framework-dependent CLI: test the apphost and installed runtime where the change matters. Source/assembly checks do not establish all packaged CLI behavior.
  • Self-contained CLI: grate.csproj enables single file publishing, native-library extraction, and partial trimming; Windows disables trimming. Exercise published binaries for resource/reflection/native-loading risks.
  • Native AOT: the inspected projects/workflows have no PublishAot lane. Self-contained does not imply native AOT. If adding/changing AOT support, publish and run an explicit AOT artifact on a supported host; existing passes provide no AOT evidence.
  • NuGet libraries: build.yml packs grate.core and all five provider packages. Existing provider tests reference projects; no dedicated packed-library consumer integration suite exists here. For package/API/resource changes, add or run a temporary consumer restoring the exact local .nupkg versions, wire a provider, and perform a real migration, checking dependency resolution, bootstrap resources, SQL effects, and history.
  • .NET tool: the build workflow packs the CLI with PackAsTool=true and SelfContained=true. skill-installation.yml instead packs SelfContained=false, installs from a local feed, and tests skill installation from that tool and a self-contained binary on Ubuntu, Windows, and macOS. This proves those skill operations, not general tool migration behavior or the build workflow’s exact tool packaging. Add consumer migration checks when those boundaries change; do not claim library-package coverage from tool tests.
  • Docker distribution: DockerGrateMigrator hardcodes gratedevs/grate with no source-specific tag or image override. It uses bind mounts, a shared test network, read-only root filesystem, and bundle extraction. GrateTestConfig__DockerImage cannot select the grate image. Align the image with the source before claiming validation; the current runner can exercise a released image. It overrides the image entrypoint, so validate entrypoint.sh separately when changed.

Matrix facts to preserve

ci.yml runs Basic_tests and all six provider suites on Ubuntu, including SqlServerCaseSensitive; build.yml omits the case-sensitive suite. Their test commands select no single TFM. Build output is not execution proof. build.yml publishes net10 self-contained RIDs win-x64, win-x86, win-arm64, linux-musl-x64, linux-musl-arm64, linux-x64, linux-arm64, osx-x64, and osx-arm64 (the last on macOS). Docker builds target linux/amd64,linux/arm64 using the Alpine/musl Dockerfile.

integration-tests.yml actively runs net10 Linux x64 self-contained CLI tests against SQL Server 2017/2019/2022/2025, PostgreSQL 12–18, MariaDB 10.4/10.10/11/12/13 RC, SQLite, and Oracle Free/XE image variants. Read its exact image tags when selecting compatibility checks. These jobs use continue-on-error: true. The grate Docker-distribution step, external/cloud test environment, and wider OS/architecture integration jobs are disabled; commented matrix entries and successful artifact publication establish no runtime coverage.

For a handoff, record source/artifact identity, selected test layers and counts, TFM, host/RID, database version/image, CLI publish form or package/image identity, commands and outcomes, skips/early returns, and remaining matrix gaps. Reuse passing evidence until a new change or unresolved risk justifies repeating it.

This site uses Just the Docs, a documentation theme for Jekyll.