For in-build plugins adding new files to the Target's source is expensive (plugin and tool-commands all run fresh)

Development Package Manager
, ,
1

My in build plugin runs from scratch every time a new file is created in Target's source directory even if it doesn't affect the plugin's input list.

About the plugin: Using swift package init --type build-tool-plugin as a starting point I made a build plugin and a CLI that imports it. The build plugin creates a Command for every text file in the Source folder. The tool in the command just creates a .swift file of the same name. Super minimum viable build plugin.

The trick is back in the CLI that imports the plugin (swift-tools-version: 5.9, vanilla swift run):

  • If I add a .swift file to the Source folder the tool still reruns for each and every text file even though .swift files shouldn't generate new commands.
  • If I just edit one of the text files the tool only runs for that text file as expected.
  • If I add a file OUTSIDE of the Source folder like a README.md the tools do not rerun.
  • If I ad a README.md in an EXCLUDED folder inside the source directory, the tools rerun.

My understanding was that in-build tools should only run if their inputs have changed or the outputs are missing. My inputs have not changed. Presumably the outputs haven't been zotted either?

I can understand rechecking when the Source directory changes, I'd rather rerun by mistake than not run. That said if it was A LOT of files / expensive processes it'd be annoying.

If I wanted to see if I could change the behavior what should I do next?

  • A: Nothing - This is the expected behavior / known issue and unavoidable / changed in 5.10, etc. :+1: Carry on.
  • B: Keep trying to twiddle the code -
    • For example: Presumably the number of .nones in the returned [Command] changes, but I tried filtering on text files in filesFromDirectory and that didn't seem to have an effect. Also presumably .nones get compactMapped out? I can try again if that's the ticket.
  • C: Look into if any of the options/flags on swift build or swift run could impact this.
  • D: File an issue.
  • E: Something else.

Thanks so much for this incredibly powerful system. Cheers.

Side Note: In an Xcode project (Just the default multi-platform template, Xcode 15.2, basic built for MacOS and iOS 15 in the simulator) the build plugin just seems to run every time, no matter what but I care less about Xcode projects over churning.

Test Projects

import PackagePlugin import Foundation @main struct FruitStoreBuild: BuildToolPlugin { /// Entry point for creating build commands for targets in Swift packages. func createBuildCommands(context: PluginContext, target: Target) async throws -> [Command] { // Find the code generator tool to run (This is what we named our actual one.). let generatorTool = try context.tool(named: "my-code-generator") // Still ensures that the target is a source module. guard let target = target as? SourceModuleTarget else { return [] } let filesToProcess = try filesFromDirectory(path: target.directory, shallow: false) // Construct a build command for each source file with a particular suffix. return filesToProcess.compactMap { createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path) } } } #if canImport(XcodeProjectPlugin) import XcodeProjectPlugin extension FruitStoreBuild: XcodeBuildToolPlugin { // Entry point for creating build commands for targets in Xcode projects. func createBuildCommands(context: XcodePluginContext, target: XcodeTarget) throws -> [Command] { // Find the code generator tool to run (replace this with the actual one). let generatorTool = try context.tool(named: "my-code-generator") // Construct a build command for each source file with a particular suffix. return target.inputFiles.map(\.path).compactMap { createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path) } } } #endif extension FruitStoreBuild { /// Shared function that returns a configured build command if the input files is one that should be processed. func createBuildCommand(for inputPath: Path, in outputDirectoryPath: Path, with generatorToolPath: Path) -> Command? { // Skip any file that doesn't have the extension we're looking for (replace this with the actual one). guard inputPath.extension == "txt" else { return .none } print("PROOF OF PLUGIN LIFE from createBuildCommand") // Return a command that will run during the build to generate the output file. let inputName = inputPath.lastComponent let outputName = inputPath.stem + ".swift" let outputPath = outputDirectoryPath.appending(outputName) return .buildCommand( displayName: "------------ Generating \(outputName) from \(inputName) ------------", executable: generatorToolPath, arguments: ["\(inputPath)", "-o", "\(outputPath)"], inputFiles: [inputPath], outputFiles: [outputPath] ) } } func filesFromDirectory(path providedPath:Path, shallow:Bool = true) throws -> [Path] { if shallow { return try FileManager.default.contentsOfDirectory(atPath: providedPath.string).compactMap { fileName in providedPath.appending([fileName]) } } else { let dataDirectoryURL = URL(fileURLWithPath: providedPath.string, isDirectory: true) var allFiles = [Path?]() let enumerator = FileManager.default.enumerator(at: dataDirectoryURL, includingPropertiesForKeys: [.isRegularFileKey], options: [.skipsHiddenFiles, .skipsPackageDescendants]) while let fileURL = enumerator?.nextObject() as? URL { if let regularFileCheck = try fileURL.resourceValues(forKeys:[.isRegularFileKey]).isRegularFile, regularFileCheck == true { allFiles.append((Path(fileURL.path()))) } } return allFiles.compactMap({$0}) } } 
import Foundation let arguments = ProcessInfo().arguments if arguments.count < 4 { print("missing arguments") } // print("ARGUMENTS") // arguments.forEach { // print($0) // } let (input, output) = (arguments[1], arguments[3]) //Added for ease of scanning for our output. print("FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE") print("from MyBuildPluginTool:", input) print("from MyBuildPluginTool:", output) var outputURL = URL(fileURLWithPath: output) let contentsOfFile = "//nothing of importance" try contentsOfFile.write(to: outputURL, atomically: true, encoding: .utf8)
2 Likes
2

Just wanted to say a million thanks for this code example!!! Pulling my hair out getting this to work. Now I have a working plugin for my TypeLocXC project. Thank you.

1 Like