Skip to content

buffer: add buffer.isUtf8 for utf8 validation #45947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Dec 25, 2022
Merged

buffer: add buffer.isUtf8 for utf8 validation #45947

merged 6 commits into from
Dec 25, 2022

Conversation

@anonrig
Copy link
Member

@anonrig anonrig commented Dec 22, 2022 •
edited
Loading

With the introduction of simdutf, we can finally have a fast way of validating utf-8 on the userland. This pull request introduces buffer.isUtf8.

This was initially requested by @KhafraDev while implementing undici websockets.
@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Dec 22, 2022
@anonrig anonrig mentioned this pull request Dec 22, 2022
Closed
4 tasks
@anonrig
Copy link
Member Author

anonrig commented Dec 22, 2022

CC @nodejs/buffer @nodejs/performance @nodejs/undici
lpinca
lpinca reviewed Dec 22, 2022
View reviewed changes
RafaelGSS
RafaelGSS reviewed Dec 22, 2022
View reviewed changes
KhafraDev
KhafraDev reviewed Dec 22, 2022
View reviewed changes
@targos
Copy link
Member

targos commented Dec 23, 2022

Can you add a test with a detached array buffer ?
lpinca
lpinca reviewed Dec 23, 2022
View reviewed changes
lpinca
lpinca reviewed Dec 23, 2022
View reviewed changes
mcollina
mcollina approved these changes Dec 23, 2022
View reviewed changes
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm
bnoordhuis
bnoordhuis reviewed Dec 23, 2022
View reviewed changes
@anonrig anonrig requested review from KhafraDev, RafaelGSS, bnoordhuis, lpinca and targos and removed request for KhafraDev December 23, 2022 17:34
@anonrig anonrig added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. notable-change PRs with changes that should be highlighted in changelogs. labels Dec 23, 2022
lpinca
lpinca approved these changes Dec 23, 2022
View reviewed changes
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 23, 2022
@github-actions github-actions bot mentioned this pull request Jan 1, 2023
Open
20 tasks
targos pushed a commit that referenced this pull request Jan 1, 2023
@anonrig @targos
buffer: add buffer.isUtf8 for utf8 validation
e445878 
PR-URL: #45947 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net>
@github-actions github-actions bot mentioned this pull request Jan 2, 2023
Open
22 tasks
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
01e1375 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: TBD
@RafaelGSS RafaelGSS mentioned this pull request Jan 2, 2023
Merged
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
fd77454 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
79cf6bc 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
85a26c6 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
3d55e4b 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
@github-actions github-actions bot mentioned this pull request Jan 3, 2023
Open
23 tasks
RafaelGSS added a commit that referenced this pull request Jan 3, 2023
@RafaelGSS
2023-01-04, Version 19.4.0 (Current)
ef55c9c 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @RafaelGSS
buffer: add buffer.isUtf8 for utf8 validation
fd33718 
PR-URL: #45947 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net>
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
980a9ca 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
6e0a7be 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
1ed0ef2 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
790cc5b 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @RafaelGSS
buffer: add buffer.isUtf8 for utf8 validation
fbe399c 
PR-URL: #45947 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net>
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
b181545 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
e58ed6d 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 6, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
22a2ec6 
Notable changes: buffer: * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 http: * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778 net * add autoSelectFamily global getter and setter (Paolo Insogna) #45777 os: * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895 util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46061
bentley1985
bentley1985 reviewed Jan 7, 2023
View reviewed changes
Copy link

@bentley1985 bentley1985 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks
juanarbol pushed a commit that referenced this pull request Jan 26, 2023
@anonrig @juanarbol
buffer: add buffer.isUtf8 for utf8 validation
907068c 
PR-URL: #45947 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net>
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
de266a4 
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: TBD
@juanarbol juanarbol mentioned this pull request Jan 28, 2023
Merged
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
85ed13d 
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
614a60d 
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803 * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 30, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
53ddd3a 
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
@anonrig @juanarbol
buffer: add buffer.isUtf8 for utf8 validation
16ee02f 
PR-URL: #45947 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Anna Henningsen <anna@addaleax.net>
juanarbol added a commit that referenced this pull request Jan 31, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
a413280 
Notable changes: * buffer * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947 * deps: * add simdutf dependency (Yagiz Nizipli) #45803 * upgrade npm to 9.1.3 (npm team) #45693 * util: * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803 PR-URL: #46396
@BethGriggs BethGriggs mentioned this pull request Feb 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs that have at least one approval, no pending requests for changes, and a CI started. buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. semver-minor PRs that contain new features and should be released in the next minor version.