I've been trying to run a performance test on a Swift package in Xcode for about an hour now, but I can't seem to get the "No baseline average for Time" message to go away. I've updated the baseline value through Xcode's UI multiple times and it just doesn't seem to stick. Is this issue unique to Swift packages or just a more general bug in Xcode?
Thanks!
The strange thing is that I can confirm setting a baseline average in Xcode adds the following two files to the Swift package:
.swiftpm/xcode/xcshareddata/xcbaselines/{Package Name}.xcbaseline/{UUID}.plist .swiftpm/xcode/xcshareddata/xcbaselines/{Package Name}.xcbaseline/Info.plist The former looks something like the following:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>classNames</key> <dict> <key>PackageTests</key> <dict> <key>testPerformance()</key> <dict> <key>com.apple.XCTPerformanceMetric_WallClockTime</key> <dict> <key>baselineAverage</key> <real>0.00903</real> <key>baselineIntegrationDisplayName</key> <string>Local Baseline</string> </dict> </dict> </dict> </dict> </dict> </plist> And here's Info.plist:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>runDestinationsByUUID</key> <dict> <key>D36CD09E-8356-41EE-9A77-1F6891B15F09</key> <dict> <key>localComputer</key> <dict> <key>busSpeedInMHz</key> <integer>…</integer> <key>cpuCount</key> <integer>…</integer> <key>cpuKind</key> <string>…</string> <key>cpuSpeedInMHz</key> <integer>…</integer> <key>logicalCPUCoresPerPackage</key> <integer>…</integer> <key>modelCode</key> <string>…</string> <key>physicalCPUCoresPerPackage</key> <integer>…</integer> <key>platformIdentifier</key> <string>com.apple.platform.macosx</string> </dict> <key>targetArchitecture</key> <string>x86_64</string> </dict> </dict> </dict> </plist> Even though Xcode automatically created these files for me, it doesn't appear to realize that it did:
Is this a known issue with how Xcode handles performance tests for Swift packages or am I just misunderstanding how they're supposed to work?
I still can't seem to get this to work. Just out of curiosity, how does everyone else run performance tests on Swift packages?
FWIW I'm having the same issue on Xcode 12.3
Thanks for confirming—I'm glad to know I'm not the only one! (For the record, I last tested this on Xcode 12.4.)
Same behaviour with Xcode 13 beta 2.
And with Xcode 13.4.
I tried both with SPM and generating an Xcode project - same issue with both, seems amnesia has hit.
My suspicion is that if Xcode is opening a package rather than a project file, it doesn't have anywhere to store the results. Hence the amnesia.
When I was wanting them retained, I'd use the command line to create an Xcode project file and use that. However that approach is now being phased out.
But Xcode is storing other stuff in .swiftpm, not sure it's the best place, but performance baselines could be there:
hassila@ice ~/G/test-release-package (main)> ls .swiftpm/xcode package.xcworkspace/ xcuserdata/ In fact, it is already being stored there. I just created a dummy package with a couple of performance tests, and in a subfolder of .swiftpm/xcode/xcshareddata/xcbaselines is a file (with random generated name) containing the following:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>classNames</key> <dict> <key>MyLibraryTests</key> <dict> <key>testCalc1()</key> <dict> <key>com.apple.XCTPerformanceMetric_WallClockTime</key> <dict> <key>baselineAverage</key> <real>0.032604</real> <key>baselineIntegrationDisplayName</key> <string>Local Baseline</string> </dict> </dict> <key>testCalc2()</key> <dict> <key>com.apple.XCTPerformanceMetric_WallClockTime</key> <dict> <key>baselineAverage</key> <real>0.033293</real> <key>baselineIntegrationDisplayName</key> <string>Local Baseline</string> </dict> </dict> </dict> </dict> </dict> </plist> The problem is even though Xcode was writing the data there, it doesn't seem to think of looking there for previous results. 
Unfortunately, the issue can be still observed in Xcode Version 14.0 beta 4.
Here is another suggestion:
Unfortunately this is still the case regarding Performance Tests in Swift Packages. Is that issue being tracked somehow and can someone point me to it? Is there an ETA for a fix?