diff --git a/README.md b/README.md index 35bd4435..58843b7e 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ Source browsing is available via: https://source.clangsharp.dev/ * [Building Managed](#building-managed) * [Building Native](#building-native) * [Generating Bindings](#generating-bindings) + * [Best practices](docs/generating-bindings-best-practices.md) * [Using locally built versions](#using-locally-built-versions) * [Spotlight](#spotlight) @@ -140,6 +141,8 @@ ClangSharpPInvokeGenerator @generate.rsp A response file allows you to specify and checkin the command line arguments in a text file, with one argument per line. For example: https://github.com/dotnet/ClangSharp/blob/main/sources/ClangSharpPInvokeGenerator/Properties/GenerateClang.rsp At a minimum, the command line expects one or more input files (`-f`), an output namespace (`-n`), and an output location (`-o`). A typical response file may also specify explicit files to traverse, configuration options, name remappings, and other fixups. +For an opinionated walkthrough of how to structure a real generation project — response-file composition, the key options and when to use them, incremental regeneration, and common pitfalls — see [Generating bindings: best practices](docs/generating-bindings-best-practices.md). + The full set of available switches: ``` ClangSharpPInvokeGenerator diff --git a/docs/generating-bindings-best-practices.md b/docs/generating-bindings-best-practices.md new file mode 100644 index 00000000..8635723c --- /dev/null +++ b/docs/generating-bindings-best-practices.md @@ -0,0 +1,271 @@ +# Generating bindings: best practices + +This guide is an opinionated walkthrough of how to drive `ClangSharpPInvokeGenerator` well for a +real, maintainable binding project. It complements the option reference in the +[main README](../README.md#generating-bindings) and the output of `--help` and `-c help`. + +If you have ever wondered *why* the generator emitted nothing, traversed half of the Windows SDK, +or produced a slightly different diff on every run, the problem is almost always in how the tool is +being invoked rather than in the tool itself. The recommendations below are drawn from +[terrafx/terrafx.interop.windows](https://github.com/terrafx/terrafx.interop.windows) — a large, +mature ClangSharp consumer maintained by the same maintainer — and from this repository's own +self-hosted [`GenerateClang.rsp`](../sources/ClangSharpPInvokeGenerator/Properties/GenerateClang.rsp). + +## Contents + +* [Structure a generation project](#structure-a-generation-project) +* [Use response files](#use-response-files) +* [Split shared settings from per-target settings](#split-shared-settings-from-per-target-settings) +* [Key options and when to use them](#key-options-and-when-to-use-them) +* [Wildcards](#wildcards) +* [Incremental regeneration](#incremental-regeneration) +* [Common pitfalls](#common-pitfalls) + +## Structure a generation project + +Treat generation as a first-class, checked-in part of your repository, not a one-off command: + +* **One folder per target.** In terrafx, `generation/` contains a folder per header being wrapped + (for example `generation/DirectX/d3d12/d3d12/`). Each folder holds three files: + * `generate.rsp` — the response file for that target, + * `header.txt` — the license/header text prefixed to every generated file (via `--headerFile`), + * an umbrella `*.h` that `#include`s exactly what the target should expose. +* **Shared configuration at the root.** Cross-cutting settings live once at `generation/` + (`settings.rsp`, `remap.rsp`, and topical `remap-*.rsp`) and are pulled in by every target. +* **A thin driver script.** terrafx's `scripts/build.ps1` simply discovers every `generate.rsp` + and regenerates each in parallel: + + ```powershell + $generateRspFiles = Get-ChildItem -Path "$generationDir" -Recurse -Filter "generate.rsp" + + $generateRspFiles | ForEach-Object -Parallel { + Push-Location -Path $_.DirectoryName + & ClangSharpPInvokeGenerator "@generate.rsp" + Pop-Location + } + ``` + + The equivalent on Linux/macOS, using `find` to discover the targets and `xargs` to run them + in parallel: + + ```bash + find "$generationDir" -name generate.rsp -print0 | + xargs -0 -P "$(nproc)" -I{} sh -c 'cd "$(dirname "$1")" && ClangSharpPInvokeGenerator "@generate.rsp"' _ {} + ``` + + The driver has no per-target knowledge — adding a new binding is just a new folder with a + `generate.rsp`. Because each target is independent, they parallelize cleanly. + +* **Check the generated code into source control** alongside the `generation/` inputs, so that + regenerating produces a reviewable diff (see [Incremental regeneration](#incremental-regeneration)). + +## Use response files + +The recommended way to invoke the generator is a response file, passed with `@`: + +``` +ClangSharpPInvokeGenerator @generate.rsp +``` + +A response file lists command-line arguments one per line, so they can be checked in and reviewed. +Two properties make them scale: + +* **`@`-includes compose.** A response file can pull in other response files by prefixing them with + `@`. terrafx's per-target `generate.rsp` begins by including the shared config, then adds only + what is unique to that target: + + ``` + @../../../settings.rsp + @../../../remap.rsp + --file + d3d12-d3d12.h + --namespace + TerraFX.Interop.DirectX + --output + ../../../../sources/Interop/Windows/DirectX/d3d12/d3d12 + ... + ``` + + `remap.rsp` is itself nothing but `@`-includes of topical files, so a single concern + (handles, namespaces, transparent structs, ...) is defined in exactly one place: + + ``` + @../../../remap-classes.rsp + @../../../remap-handles.rsp + @../../../remap-namespaces.rsp + @../../../remap-transparent.rsp + @../../../remap-types.rsp + ... + ``` + +* **Relative paths resolve from the invoking directory.** Because the driver changes into each + target folder before invoking the tool, every relative path (`--file`, `--output`, + `--include-directory`, and the `@`-includes) is written relative to that folder. + +For a minimal single-file example, see this repository's own +[`GenerateClang.rsp`](../sources/ClangSharpPInvokeGenerator/Properties/GenerateClang.rsp). + +## Split shared settings from per-target settings + +Decide where each option belongs by asking whether it is a property of *the whole project* or of +*one target*. + +**Shared (`settings.rsp`)** — the things that should be identical everywhere: + +* `--additional` clang arguments, including the architecture (`-m64`) and warning suppression + (`-Wno-comment`, `-Wno-deprecated-declarations`, `-Wno-ignored-attributes`, ...). Suppressing the + noise up front keeps real diagnostics visible. +* The `--config` flag set (see [config flags](#config-flags--c)). +* `--define-macro` values that shape what the headers expose (for example `UNICODE`, `INITGUID`). +* `--include-directory` search paths (your local headers plus the SDK). +* `--headerFile header.txt` and project-wide conventions such as `--with-callconv *=Winapi`. + +**Per-target (`generate.rsp`)** — the things unique to one header: + +* `--file` (the umbrella header) and `--traverse` (which included files to emit). +* `--namespace`, `--methodClassName`, `--output`, `--test-output`. +* Target-scoped `--exclude`, `--with-attribute`, and `--with-librarypath`. + +Keeping remappings in shared topical files means a type like `HANDLE` or a namespace like `ABM` is +mapped once and applied consistently to every target that encounters it. + +## Key options and when to use them + +The switch reference in the [README](../README.md#generating-bindings), `--help`, and `-c help` are +authoritative. This section groups the options by intent and notes when to reach for each. + +### Inputs and traversal + +* **`-f, --file`** — the input header(s) to parse. Usually a single umbrella header per target. +* **`-t, --traverse`** — *which* included files should have bindings emitted. Without it, the + generator will happily emit everything transitively reachable, including system headers. This is + the single most important option for keeping output scoped. +* **`-I, --include-directory`** — the header search path. Order matters; put your own headers before + the SDK so local overrides win. +* **`-D, --define-macro`** — define macros before parsing (for example `INITGUID`, `UNICODE`). +* **`-a, --additional`** — pass raw arguments to clang. Use for the target architecture and for + `-Wno-*` warning suppression. +* **`-x, --language` / `-std, --std`** — force C vs C++ and the language standard when the headers + need it. + +### Naming and remapping + +* **`-r, --remap`** — rename a declaration during generation. The topical `remap-*.rsp` files show + the pattern: opaque handle typedefs remapped to themselves to keep a distinct C# type, and + pointer-sized integers remapped to `@nint`/`@nuint` (`SIZE_T=@nuint`, `INT_PTR=@nint`). +* **`-rt, --remap-type` / `-rf, --remap-field`** — take precedence over `--remap` and disambiguate + the case where a type and a field share a name. +* **`-wn, --with-namespace`** — place a declaration (or a whole prefix family such as `ABM=`, + `ACCESS=`) into a chosen namespace. terrafx funnels many prefixes into + `TerraFX.Interop.Windows`. +* **`-wu, --with-using`, `-wc, --with-class`, `-m, --methodClassName`, `-p, --prefixStrip`** — + control the `using` directives, the owning static class for free functions, and prefix stripping + (for example stripping `clang_` in this repo's own bindings). + +### Type shaping + +* **`-wts, --with-transparent-struct`** — wrap a typedef in a strongly-typed transparent struct. + The value is `underlying;kind`, and the *kind* controls the generated conveniences — + `HANDLE=void*;HandleWin32`, `BOOL=int;Boolean`, `COLORREF=uint;TypedefHex`. This is how terrafx + turns dozens of `void*` handle typedefs into distinct, safe C# types. +* **`-wt, --with-type`** — set the backing type for an enum declaration. +* **`-wa, --with-attribute`** — attach an attribute to a declaration. Two common uses: marking flag + enums (`D3D12_RESOURCE_FLAGS=Flags`) and platform gating + (`ID3D12Device4=SupportedOSPlatform("windows10.0.19043.0")`). +* **`-was, --with-access-specifier`, `-wp, --with-packing`, `-wl, --with-length`, + `-wg, --with-guid`, `-wro, --with-readonly`** — fine-grained control over accessibility, struct + packing, array lengths, GUIDs, and `readonly` methods. + +### Interop and P/Invoke + +* **`-l, --libraryPath` / `-wlb, --with-librarypath`** — the DLL name for `DllImport`. Use + `--with-librarypath` to route specific functions to specific libraries + (`D3D12CreateDevice=d3d12`). +* **`-wcc, --with-callconv`** — the calling convention, commonly `*=Winapi` project-wide. +* **`-wsle, --with-setlasterror`, `-wsgct, --with-suppressgctransition`, + `-wmi, --with-manual-import`** — opt specific functions into `SetLastError`, + `SuppressGCTransition` (for tiny, hot, non-blocking calls such as the D2D1 math helpers), or + manual import. + +### Exclusions + +* **`-e, --exclude` / `-i, --include`** — drop or keep specific declarations. Use `--exclude` for + macros and helpers that don't translate (for example the `D3D12_DECODE_*`/`D3D12_ENCODE_*` macro + helpers, or `CINDEX_VERSION_*` in this repo). +* The `exclude-*` [config flags](#config-flags--c) handle *categories* of declarations + (COM proxies, empty records, enum operators, ...). + +### Config flags (`-c`) + +`-c, --config` toggles generation behavior. Run `ClangSharpPInvokeGenerator -c help` for the full, +authoritative list. The groups worth knowing: + +* **Codegen level** — `compatible-codegen` (.NET Standard 2.0), `default-codegen` (current LTS), + `latest-codegen`, `preview-codegen`. Pick the one that matches your target framework; the more + modern levels enable function pointers and other niceties. +* **File layout** — `single-file` (default) vs `multi-file` (roughly one type per file, easier to + review and diff for large projects). +* **Type assumptions** — `windows-types` vs `unix-types`. +* **Vtbl strategy** — `explicit-vtbls`, `implicit-vtbls` (default), `trimmable-vtbls`. +* **Tests** — `generate-tests-nunit` / `generate-tests-xunit` emit size/blittability tests + alongside the bindings. +* **Generation toggles** — for example `generate-aggressive-inlining`, + `generate-file-scoped-namespaces`, `generate-guid-member`, `generate-marker-interfaces`, + `generate-native-inheritance-attribute`, `generate-vtbl-index-attribute`. + +### Output and test modes + +* **`-o, --output`** — where bindings are written. With `multi-file`, this is a directory. +* **`-to, --test-output`** — where the generated tests are written (pair with a + `generate-tests-*` config flag). +* **`-om, --output-mode`** — `CSharp` (default) or `Xml`. + +## Wildcards + +The `--with-*` options accept `*` as a catch-all. This is how project-wide conventions are applied +in one line, for example: + +``` +--with-callconv +*=Winapi +``` + +``` +--with-access-specifier +*=Internal +``` + +## Incremental regeneration + +* **Commit the generated output.** With generated code in source control, regenerating turns every + header/SDK bump into a reviewable diff instead of an opaque black box. +* **Regenerate per target, in parallel.** Because each `generate.rsp` is self-contained, the driver + can regenerate every target concurrently; you only re-run the ones whose inputs changed. +* **Pin your inputs.** Keep the SDK/include versions in the shared `settings.rsp` (terrafx pins an + explicit Windows Kits version) so output stays deterministic across machines and CI. +* **Diagnose with the `log-*` flags** rather than guessing. These are the practical answer when the + output isn't what you expected: + * `log-exclusions` — lists every declaration that was excluded, and whether it matched exactly or + partially. Use this when something you wanted is missing. + * `log-potential-typedef-remappings` — surfaces typedefs that look like they *should* be remapped, + which helps you find missing `--remap`/`--with-transparent-struct` entries. + * `log-visited-files` — lists the files that were actually visited, which is the fastest way to + catch a `--traverse`/`--include-directory` mistake. + +## Common pitfalls + +* **Forgetting `--traverse`.** Without it the generator emits everything reachable, including system + headers. Scope every target with `--traverse` and confirm with `log-visited-files`. +* **`--include-directory` order.** If the wrong copy of a header is found first, you get subtly + wrong output. Put local include directories before the SDK. +* **Skipping warning suppression.** Real headers produce many benign clang warnings; add the + `-Wno-*` set to `--additional` so genuine parse errors aren't buried. +* **`--remap` vs `--remap-type`/`--remap-field` clashes.** When a type and field share a name, use + the more specific option — it takes precedence and avoids remapping the wrong entity. +* **Transparent-struct kind strings.** The kind after the `;` (for example `HandleWin32`, + `Boolean`, `TypedefHex`) determines the generated helpers; picking the wrong one changes the API + surface. Model new handles on the existing `remap-transparent.rsp` entries. +* **Flag enums.** Native enums used as bit flags won't get `[Flags]` automatically; add + `--with-attribute NAME=Flags`. +* **Platform-specific paths.** Absolute SDK include paths in `settings.rsp` are environment + specific; keep them centralized so there is a single place to update them. diff --git a/sources/ClangSharpPInvokeGenerator/CommandLine.cs b/sources/ClangSharpPInvokeGenerator/CommandLine.cs index 41442b46..410c86c9 100644 --- a/sources/ClangSharpPInvokeGenerator/CommandLine.cs +++ b/sources/ClangSharpPInvokeGenerator/CommandLine.cs @@ -241,7 +241,7 @@ private void ExpandToken(string token, List result, int depth) } } - public void WriteHelp(TextWriter writer, string name, string description, string epilogTitle, string epilog) + public void WriteHelp(TextWriter writer, string name, string description, string epilogTitle, string epilog, string moreInfoTitle, string moreInfo) { writer.WriteLine(name); writer.WriteLine($" {description}"); @@ -277,6 +277,9 @@ public void WriteHelp(TextWriter writer, string name, string description, string writer.WriteLine(); writer.WriteLine(epilogTitle); writer.WriteLine(epilog); + writer.WriteLine(); + writer.WriteLine(moreInfoTitle); + writer.WriteLine(moreInfo); } public static void WriteOptionHelp(TextWriter writer, CommandLineOption option, IReadOnlyList rows) diff --git a/sources/ClangSharpPInvokeGenerator/Program.cs b/sources/ClangSharpPInvokeGenerator/Program.cs index fc8bd19d..e9f2dd4a 100644 --- a/sources/ClangSharpPInvokeGenerator/Program.cs +++ b/sources/ClangSharpPInvokeGenerator/Program.cs @@ -26,13 +26,16 @@ internal static partial class Program private const string WildcardsTitle = "Wildcards:"; private const string Wildcards = "You can use * as catch-all rule for remapping procedures. For example if you want make all of your generated code internal you can use --with-access-specifier *=Internal."; + private const string MoreInfoTitle = "More information:"; + private const string MoreInfo = "See https://github.com/dotnet/ClangSharp/blob/main/docs/generating-bindings-best-practices.md for a guide on structuring a generation project and using these options."; + public static int Main(params string[] args) { s_parser.Parse(args); if (s_helpOption.IsPresent) { - s_parser.WriteHelp(Console.Out, Name, Description, WildcardsTitle, Wildcards); + s_parser.WriteHelp(Console.Out, Name, Description, WildcardsTitle, Wildcards, MoreInfoTitle, MoreInfo); return 0; } @@ -511,7 +514,7 @@ public static int Run() } Console.Error.Write(Environment.NewLine); - s_parser.WriteHelp(Console.Out, Name, Description, WildcardsTitle, Wildcards); + s_parser.WriteHelp(Console.Out, Name, Description, WildcardsTitle, Wildcards, MoreInfoTitle, MoreInfo); return -1; } diff --git a/sources/ClangSharpPInvokeGenerator/README.md b/sources/ClangSharpPInvokeGenerator/README.md index b8befd84..410b9735 100644 --- a/sources/ClangSharpPInvokeGenerator/README.md +++ b/sources/ClangSharpPInvokeGenerator/README.md @@ -32,6 +32,7 @@ You can see the full set of available switches by passing `--help`, and the avai ## More information +* Best practices for driving the generator: https://github.com/dotnet/ClangSharp/blob/main/docs/generating-bindings-best-practices.md * Full documentation and the list of options: https://github.com/dotnet/ClangSharp#generating-bindings * Source repository: https://github.com/dotnet/ClangSharp * Nightly packages: https://pkgs.clangsharp.dev/index.json