Parley Benchmarks #405

taj-p · 2025-08-25T22:32:27Z

Adds in an initial set of benchmarks to aid Parley development.

I've opted to use Tango because I like the quick development loop. We can run 20 benchmark cases in ~10 seconds. It also "feels" more reliable.

Here's what its output looks like (results from an upcoming caching PR).

Extra

Moved font assets to parley_dev
Created text samples for use in benchmarks and around Parley in parley_dev/assets/text_samples

cc @conor-93

xorgy

Looks good and runs for me. I get a roughly ± 2 % noise floor (comparing this branch to itself in Tango) when running with a pinned CPU and quieted IRQs on the thread... which is not too bad!
I settled on Divan recently for one of my things, because the less-sophisticated statistical analysis and research-driven sampling method helped a lot with certain types of micro benchmarks that were completely impossible to measure reliably with Criterion (due to a weird sampling behavior, or maybe Criterion was somehow poisoning the branch predictor for this particular function.. unsure).
Tango is very cool, I like the differential paired timing analysis, reminds me of balanced signal pairs/differential signalling. :+ )

DJMcNab

There is one blocker, which is updating the file paths for the fonts in the root README (and also removing them from parley/README.md)

parley_bench/.gitignore

DJMcNab · 2025-08-26T07:59:10Z

parley_bench/build.rs

+
+//! # Parley Bench Build Script
+
+fn main() {


Presumably, this is the boilerplate required by Tango? We should probably comment that there

Yup! Added in cf13c64

parley_bench/README.md

DJMcNab · 2025-08-26T08:02:38Z

parley_bench/README.md

@@ -0,0 +1,19 @@
+# Parley Benchmarks
+
+A suite of benchmarks used to evaluate Parley performance.


We should link to Tango somewhere here, so people have a chance to understand what's actually going on.

It looks like the right link is https://github.com/bazhenov/tango. Something like:

This uses [Tango](https://github.com/bazhenov/tango) to perform paired benchmarking.

I also just saw that Tango would require manual changes and nightly on Windows - we should also document that (even if it's just "Windows might require additional setup, see Tango's docs for details")

Thank you! Added in 4d5a16b

DJMcNab · 2025-08-26T08:34:40Z

parley_dev/assets/text_samples/README.md

@@ -0,0 +1,11 @@
+# Sample Text Attribution
+
+All samples are licensed under the [Creative Commons Attribution-ShareAlike License](https://creativecommons.org/licenses/by-sa/4.0/) as copied to ./LICENSE.


Wikisource documents that Moby Dick is in the public domain (see the note at the bottom of https://en.wikisource.org/wiki/Moby-Dick_(1851)_US_edition).
It does the same for the Tosa Diary source - if you see the note at the bottom of that source, it states that it's in the public domain in the US. We are already relying on this public domain status for them to be licensed under the license which we claim, and I believe that the intention of Wikisource's license is to cover all the copyrightable content (e.g. the intro at the top of the Moby Dick page).

The situation for Al-Kindi - First Philosophy is slightly less clear, but given that the texts were written more than 1000 years ago, I imagine the situation is similar. I've not found any sources which discuss this (in English).

As such, I think we have two pathways here:

We can either just use them without copying in the license, still mentioning that we got them from Wikisource. Since they are in the public domain. This is the direction I'd choose, if you agree with my analysis.

Use this form, which also requires us to mention that these files are licensed as discussed in the root README. That is, this is a step which hasn't been done in this PR; the text at https://github.com/linebender/parley?tab=readme-ov-file#license would need to be updated.

Yes, I agree with your assessment. I've made this change in e5f2e46

DJMcNab · 2025-08-26T08:35:38Z

parley_dev/src/lib.rs

+#[derive(Debug)]
+pub struct Sample {
+ /// The name of the sample.
+ pub name: &'static str,


I'd be tempted to also add a "title" field here, although I'm not sure where we'd use it.

Added in 385e49c

DJMcNab · 2025-08-26T08:36:31Z

parley_dev/assets/fonts/arimo_fonts/Arimo-VariableFont_wght.ttf

As this file has moved, it's reference in the root README needs to be updated. This is blocking

Nice catch! Thank you - fixed in 711c011

DJMcNab · 2025-08-26T08:40:24Z

parley_dev/Cargo.toml

I'd like for this folder to have a README. It should have an intro:

Data and fonts used in Parley's tests and benchmarks.

and something like:

Some files included in this folder are under different licenses to the rest of this workspace: - The font file `Arimo-VariableFont_wght.ttf` in `./assets/arimo_fonts/` is licensed solely as documented in that folder (and is licensed under the Apache License, Version 2.0). - The font file `Roboto-Regular.ttf` in `./assets/roboto_fonts/` is licensed solely as documented in that folder (and is licensed under the Apache License, Version 2.0). - The font file `NotoKufiArabic-Regular.otf` in `./assets/noto_fonts/` is licensed solely as documented in that folder (and is licensed under the SIL Open Font License, Version 1.1).

Great idea - implemented in 711c011

DJMcNab · 2025-08-26T09:28:31Z

By the way, when I spoke to Denis at RustWeek, they mentioned that they have had some success using this technique on GitHub's CI runners (~10% noise floor). It could be worth experimenting with automatically running some of these benchmarks, and somehow reporting results (they need only be used as a signal to trigger manual benchmarking)

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

…ajp/microbenchmarks

DJMcNab

The most important suggestion here is updating the content of the Parley package's README. Approving if and only if that's fixed (to clarify, this is because the file mentioned in the "License" section is no longer a descendent of that folder).

parley_dev/assets/text_samples/README.md

DJMcNab · 2025-08-27T09:13:51Z

parley_dev/assets/text_samples/README.md

+
+| Sample | Title | Amendments | Source |
+| ----------------- | ---------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `./arabic.txt` | Al-Kindi - First Philosophy | None | [Wiki Source](https://ar.wikisource.org/wiki/%D8%A7%D9%84%D9%83%D9%86%D8%AF%D9%8A_-_%D8%A7%D9%84%D9%81%D9%84%D8%B3%D9%81%D8%A9_%D8%A7%D9%84%D8%A3%D9%88%D9%84%D9%89) |


Maximum nit: Wikisource tends to be stylised "Wikisource" instead of "Wiki Source". This is non-blocking

Removed in ca1845a

CHANGELOG.md

DJMcNab · 2025-08-27T09:17:37Z

README.md

- The font file `Arimo-VariableFont_wght.ttf` in `/parley/tests/assets/arimo_fonts/` is licensed solely as documented in that folder (and is licensed under the Apache License, Version 2.0).
- The font file `Roboto-Regular.ttf` in `/parley/tests/assets/roboto_fonts/` is licensed solely as documented in that folder (and is licensed under the Apache License, Version 2.0).
- The font file `NotoKufiArabic-Regular.otf` in `/parley/tests/assets/noto_fonts/` is licensed solely as documented in that folder (and is licensed under the SIL Open Font License, Version 1.1).
+Some files used for tests and benchmarks are under different licenses. See `./parley_dev/README.md` for details.


This still needs to be reflected in ./parley/README.md (i.e. the "Some files used for tests are under different licenses:" section can just be removed).

🤦 - Thanks for this catch. Removed in 2e61fa6

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

…..."

…ajp/microbenchmarks

Was removed in linebender#405

Was removed in #405

xorgy · 2025-08-27T13:31:12Z

By the way, when I spoke to Denis at RustWeek, they mentioned that they have had some success using this technique on GitHub's CI runners (~10% noise floor). It could be worth experimenting with automatically running some of these benchmarks, and somehow reporting results (they need only be used as a signal to trigger manual benchmarking)

One thing I've noticed is that the Github arm runners (not sure how long they are offering these specific ones for public open source repos but) have lower variance, and are faster than their cheap public x86 runners. Here are two equivalent benchmarks (same code, one is a squashed version of the other):

https://github.com/xorgy/joto/actions/runs/17196848204/job/48780185572#step:5:46
https://github.com/xorgy/joto/actions/runs/17196835034/job/48780150878#step:5:46

10% is about right I think. It is good for a sanity check. I noticed that regardless of whether I'm using criterion, tango, or divan, LTO and constraining the codegen units have helped make results more useful. (seems to lead to more similar layouts between builds, even when the implementation changes, but just because I see that in the disassembly doesn't mean it's why I see more consistency). Only catch for LTO is that applying it to any bench built with these requires nightly for a number of fun reasons.

It'd be very cool if somebody somewhere was working on a port of the Stabilizer technique to Rust. :+ )

@conor-93

Caches HarfRust `ShapeData`, `ShaperInstance`, and `ShapePlan`s using an LRU inspired by Swash's implementation. Using #405 to measure, the performance improvement is very good. <img width="653" height="279" alt="out" src="https://github.com/user-attachments/assets/d157bb9c-e063-44e2-85be-afea7b242778" /> cc @conor-93

taj-p added 30 commits August 23, 2025 07:15

init commit

a1babad

.

57aa011

.

d4883b4

Fix cluster info selection + add more tests

e33dc3a

.

aa5d351

.

94c0cdc

.

c8506d2

.

b08c014

.

fc08ffc

.

c9b862a

.

a17e2fe

.

3658a9f

.

5008b69

Add CHANGELOG

f50f934

Merge branch 'tajp/moreTestsAndFixClusterInfo' into tajp/tango

84c85f9

Move assets to parley_dev

8c602ce

Add styled benchmark

f80078b

Use smaller sample text

da896d8

Update text samples README

4ffb01a

.

1285574

.

27cd3c3

Merge remote-tracking branch 'origin/main' into tajp/microbenchmarks

5ee1704

.

d5d7108

.

5c4c8d4

.

b4e6d1e

.

e93fe11

.

930c61e

.

a150ab4

.

c15ccc0

.

b822062

xorgy approved these changes Aug 26, 2025

View reviewed changes

DJMcNab requested changes Aug 26, 2025

View reviewed changes

taj-p and others added 9 commits August 27, 2025 18:15

Update parley_bench/.gitignore

eea2340

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

build.rs docs

cf13c64

Update parley_bench/README.md

8189534

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

Merge branch 'tajp/microbenchmarks' of github.com:taj-p/parley into t…

86ce3de

…ajp/microbenchmarks

README updates

4d5a16b

Remove text samples license

e5f2e46

Add title to Sample

385e49c

Update root and parley dev README

711c011

.

17731d2

taj-p requested a review from DJMcNab August 27, 2025 09:01

DJMcNab approved these changes Aug 27, 2025

View reviewed changes

taj-p and others added 5 commits August 27, 2025 19:07

Update parley_dev/assets/text_samples/README.md

fb39129

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

Update CHANGELOG.md

d6f709f

Co-authored-by: Daniel McNab <36049421+DJMcNab@users.noreply.github.com>

Remove parley/README section starting with "Some files used for tests…

2e61fa6

…..."

Merge branch 'tajp/microbenchmarks' of github.com:taj-p/parley into t…

c9788be

…ajp/microbenchmarks

s/Wiki Source/Wikisource/g

ca1845a

taj-p enabled auto-merge August 27, 2025 09:45

taj-p added this pull request to the merge queue Aug 27, 2025

Merged via the queue into linebender:main with commit 2226403 Aug 27, 2025
24 checks passed

taj-p deleted the tajp/microbenchmarks branch August 27, 2025 09:54

DJMcNab mentioned this pull request Aug 27, 2025

Restore the Rust Code Of Conduct link #407

Merged

DJMcNab added a commit to DJMcNab/parley that referenced this pull request Aug 27, 2025

Restore the Rust Code Of Conduct link

90feaa3

Was removed in linebender#405

github-merge-queue bot pushed a commit that referenced this pull request Aug 27, 2025

Restore the Rust Code Of Conduct link (#407)

46e52d6

Was removed in #405

DJMcNab mentioned this pull request Oct 6, 2025

Prepare for 0.6.0 release #426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Parley Benchmarks #405

Parley Benchmarks #405

Uh oh!

taj-p commented Aug 25, 2025 •

edited

Loading

xorgy left a comment •

edited

Loading

DJMcNab left a comment

Uh oh!

DJMcNab Aug 26, 2025

taj-p Aug 27, 2025

Uh oh!

DJMcNab Aug 26, 2025

DJMcNab Aug 27, 2025

taj-p Aug 27, 2025

DJMcNab Aug 26, 2025 •

edited

Loading

taj-p Aug 27, 2025

DJMcNab Aug 26, 2025

taj-p Aug 27, 2025

DJMcNab Aug 26, 2025

taj-p Aug 27, 2025

DJMcNab Aug 26, 2025

taj-p Aug 27, 2025

DJMcNab commented Aug 26, 2025

DJMcNab left a comment

Uh oh!

DJMcNab Aug 27, 2025

taj-p Aug 27, 2025

Uh oh!

DJMcNab Aug 27, 2025

taj-p Aug 27, 2025

Uh oh!

xorgy commented Aug 27, 2025 •

edited

Loading

Labels

3 participants

		@@ -0,0 +1,19 @@
		# Parley Benchmarks

		A suite of benchmarks used to evaluate Parley performance.

		@@ -0,0 +1,11 @@
		# Sample Text Attribution

		All samples are licensed under the [Creative Commons Attribution-ShareAlike License](https://creativecommons.org/licenses/by-sa/4.0/) as copied to ./LICENSE.

Parley Benchmarks #405

Parley Benchmarks #405

Uh oh!

Conversation

taj-p commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Extra

xorgy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

DJMcNab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DJMcNab Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DJMcNab commented Aug 26, 2025

DJMcNab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

xorgy commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

3 participants

taj-p commented Aug 25, 2025 •

edited

Loading

xorgy left a comment •

edited

Loading

DJMcNab Aug 26, 2025 •

edited

Loading

xorgy commented Aug 27, 2025 •

edited

Loading