| Copyright | (c) 2009, 2010, 2012 Bryan O'Sullivan |
|---|---|
| License | BSD-style |
| Maintainer | bos@serpentine.com |
| Stability | experimental |
| Portability | GHC |
| Safe Haskell | Trustworthy |
| Language | Haskell98 |
Data.Text.Lazy
Contents
Description
A time and space-efficient implementation of Unicode text using lists of packed arrays.
Note: Read below the synopsis for important notes on the use of this module.
The representation used by this module is suitable for high performance use and for streaming large quantities of data. It provides a means to manipulate a large body of text without requiring that the entire content be resident in memory.
Some operations, such as concat, append, reverse and cons, have better time complexity than their Data.Text equivalents, due to the underlying representation being a list of chunks. For other operations, lazy Texts are usually within a few percent of strict ones, but often with better heap usage if used in a streaming fashion. For data larger than available memory, or if you have tight memory constraints, this module will be the only option.
This module is intended to be imported qualified, to avoid name clashes with Prelude functions. eg.
import qualified Data.Text.Lazy as L
- data Text
- pack :: String -> Text
- unpack :: Text -> String
- singleton :: Char -> Text
- empty :: Text
- fromChunks :: [Text] -> Text
- toChunks :: Text -> [Text]
- toStrict :: Text -> Text
- fromStrict :: Text -> Text
- foldrChunks :: (Text -> a -> a) -> a -> Text -> a
- foldlChunks :: (a -> Text -> a) -> a -> Text -> a
- cons :: Char -> Text -> Text
- snoc :: Text -> Char -> Text
- append :: Text -> Text -> Text
- uncons :: Text -> Maybe (Char, Text)
- head :: Text -> Char
- last :: Text -> Char
- tail :: Text -> Text
- init :: Text -> Text
- null :: Text -> Bool
- length :: Text -> Int64
- compareLength :: Text -> Int64 -> Ordering
- map :: (Char -> Char) -> Text -> Text
- intercalate :: Text -> [Text] -> Text
- intersperse :: Char -> Text -> Text
- transpose :: [Text] -> [Text]
- reverse :: Text -> Text
- replace :: Text -> Text -> Text -> Text
- toCaseFold :: Text -> Text
- toLower :: Text -> Text
- toUpper :: Text -> Text
- toTitle :: Text -> Text
- justifyLeft :: Int64 -> Char -> Text -> Text
- justifyRight :: Int64 -> Char -> Text -> Text
- center :: Int64 -> Char -> Text -> Text
- foldl :: (a -> Char -> a) -> a -> Text -> a
- foldl' :: (a -> Char -> a) -> a -> Text -> a
- foldl1 :: (Char -> Char -> Char) -> Text -> Char
- foldl1' :: (Char -> Char -> Char) -> Text -> Char
- foldr :: (Char -> a -> a) -> a -> Text -> a
- foldr1 :: (Char -> Char -> Char) -> Text -> Char
- concat :: [Text] -> Text
- concatMap :: (Char -> Text) -> Text -> Text
- any :: (Char -> Bool) -> Text -> Bool
- all :: (Char -> Bool) -> Text -> Bool
- maximum :: Text -> Char
- minimum :: Text -> Char
- scanl :: (Char -> Char -> Char) -> Char -> Text -> Text
- scanl1 :: (Char -> Char -> Char) -> Text -> Text
- scanr :: (Char -> Char -> Char) -> Char -> Text -> Text
- scanr1 :: (Char -> Char -> Char) -> Text -> Text
- mapAccumL :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)
- mapAccumR :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)
- repeat :: Char -> Text
- replicate :: Int64 -> Text -> Text
- cycle :: Text -> Text
- iterate :: (Char -> Char) -> Char -> Text
- unfoldr :: (a -> Maybe (Char, a)) -> a -> Text
- unfoldrN :: Int64 -> (a -> Maybe (Char, a)) -> a -> Text
- take :: Int64 -> Text -> Text
- takeEnd :: Int64 -> Text -> Text
- drop :: Int64 -> Text -> Text
- dropEnd :: Int64 -> Text -> Text
- takeWhile :: (Char -> Bool) -> Text -> Text
- dropWhile :: (Char -> Bool) -> Text -> Text
- dropWhileEnd :: (Char -> Bool) -> Text -> Text
- dropAround :: (Char -> Bool) -> Text -> Text
- strip :: Text -> Text
- stripStart :: Text -> Text
- stripEnd :: Text -> Text
- splitAt :: Int64 -> Text -> (Text, Text)
- span :: (Char -> Bool) -> Text -> (Text, Text)
- breakOn :: Text -> Text -> (Text, Text)
- breakOnEnd :: Text -> Text -> (Text, Text)
- break :: (Char -> Bool) -> Text -> (Text, Text)
- group :: Text -> [Text]
- groupBy :: (Char -> Char -> Bool) -> Text -> [Text]
- inits :: Text -> [Text]
- tails :: Text -> [Text]
- splitOn :: Text -> Text -> [Text]
- split :: (Char -> Bool) -> Text -> [Text]
- chunksOf :: Int64 -> Text -> [Text]
- lines :: Text -> [Text]
- words :: Text -> [Text]
- unlines :: [Text] -> Text
- unwords :: [Text] -> Text
- isPrefixOf :: Text -> Text -> Bool
- isSuffixOf :: Text -> Text -> Bool
- isInfixOf :: Text -> Text -> Bool
- stripPrefix :: Text -> Text -> Maybe Text
- stripSuffix :: Text -> Text -> Maybe Text
- commonPrefixes :: Text -> Text -> Maybe (Text, Text, Text)
- filter :: (Char -> Bool) -> Text -> Text
- find :: (Char -> Bool) -> Text -> Maybe Char
- breakOnAll :: Text -> Text -> [(Text, Text)]
- partition :: (Char -> Bool) -> Text -> (Text, Text)
- index :: Text -> Int64 -> Char
- count :: Text -> Text -> Int64
- zip :: Text -> Text -> [(Char, Char)]
- zipWith :: (Char -> Char -> Char) -> Text -> Text -> Text
Fusion
Most of the functions in this module are subject to fusion, meaning that a pipeline of such functions will usually allocate at most one Text value.
As an example, consider the following pipeline:
import Data.Text.Lazy as T import Data.Text.Lazy.Encoding as E import Data.ByteString.Lazy (ByteString) countChars :: ByteString -> Int countChars = T.length . T.toUpper . E.decodeUtf8
From the type signatures involved, this looks like it should allocate one ByteString value, and two Text values. However, when a module is compiled with optimisation enabled under GHC, the two intermediate Text values will be optimised away, and the function will be compiled down to a single loop over the source ByteString.
Functions that can be fused by the compiler are documented with the phrase "Subject to fusion".
Acceptable data
A Text value is a sequence of Unicode scalar values, as defined in §3.9, definition D76 of the Unicode 5.2 standard: http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf#page=35. As such, a Text cannot contain values in the range U+D800 to U+DFFF inclusive. Haskell implementations admit all Unicode code points (§3.4, definition D10) as Char values, including code points from this invalid range. This means that there are some Char values that are not valid Unicode scalar values, and the functions in this module must handle those cases.
Within this module, many functions construct a Text from one or more Char values. Those functions will substitute Char values that are not valid Unicode scalar values with the replacement character "�" (U+FFFD). Functions that perform this inspection and replacement are documented with the phrase "Performs replacement on invalid scalar values".
(One reason for this policy of replacement is that internally, a Text value is represented as packed UTF-16 data. Values in the range U+D800 through U+DFFF are used by UTF-16 to denote surrogate code points, and so cannot be represented. The functions replace invalid scalar values, instead of dropping them, as a security measure. For details, see Unicode Technical Report 36, §3.5: http://unicode.org/reports/tr36#Deletion_of_Noncharacters)
Types
Instances
| IsList Text | |
| Eq Text | |
| Data Text | This instance preserves data abstraction at the cost of inefficiency. We omit reflection services for the sake of data abstraction. This instance was created by copying the updated behavior of |
| Ord Text | |
| Read Text | |
| Show Text | |
| IsString Text | |
| Monoid Text | |
| Binary Text | |
| NFData Text | |
| Typeable * Text | |
| type Item Text = Char |
Creation and elimination
singleton :: Char -> Text Source
O(1) Convert a character into a Text. Subject to fusion. Performs replacement on invalid scalar values.
foldrChunks :: (Text -> a -> a) -> a -> Text -> a Source
Consume the chunks of a lazy Text with a natural right fold.
foldlChunks :: (a -> Text -> a) -> a -> Text -> a Source
Consume the chunks of a lazy Text with a strict, tail-recursive, accumulating left fold.
Basic interface
cons :: Char -> Text -> Text infixr 5 Source
O(n) Adds a character to the front of a Text. This function is more costly than its List counterpart because it requires copying a new array. Subject to fusion.
snoc :: Text -> Char -> Text Source
O(n) Adds a character to the end of a Text. This copies the entire array in the process, unless fused. Subject to fusion.
O(1) Returns the first character of a Text, which must be non-empty. Subject to fusion.
O(1) Returns the last character of a Text, which must be non-empty. Subject to fusion.
O(1) Returns all characters after the head of a Text, which must be non-empty. Subject to fusion.
O(1) Returns all but the last character of a Text, which must be non-empty. Subject to fusion.
compareLength :: Text -> Int64 -> Ordering Source
Transformations
intercalate :: Text -> [Text] -> Text Source
O(n) The intercalate function takes a Text and a list of Texts and concatenates the list after interspersing the first argument between each element of the list.
intersperse :: Char -> Text -> Text Source
O(n) The intersperse function takes a character and places it between the characters of a Text. Subject to fusion. Performs replacement on invalid scalar values.
Arguments
| :: Text |
|
| -> Text |
|
| -> Text |
|
| -> Text |
O(m+n) Replace every non-overlapping occurrence of needle in haystack with replacement.
This function behaves as though it was defined as follows:
replace needle replacement haystack =intercalatereplacement (splitOnneedle haystack)
As this suggests, each occurrence is replaced exactly once. So if needle occurs in replacement, that occurrence will not itself be replaced recursively:
replace "oo" "foo" "oo" == "foo"
In cases where several instances of needle overlap, only the first one will be replaced:
replace "ofo" "bar" "ofofo" == "barfo"
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
Case conversion
With Unicode text, it is incorrect to use combinators like map toUpper to case convert each character of a string individually. Instead, use the whole-string case conversion functions from this module. For correctness in different writing systems, these functions may map one input character to two or three output characters.
toCaseFold :: Text -> Text Source
O(n) Convert a string to folded case. Subject to fusion.
This function is mainly useful for performing caseless (or case insensitive) string comparisons.
A string x is a caseless match for a string y if and only if:
toCaseFold x == toCaseFold y
The result string may be longer than the input string, and may differ from applying toLower to the input string. For instance, the Armenian small ligature men now (U+FB13) is case folded to the bigram men now (U+0574 U+0576), while the micro sign (U+00B5) is case folded to the Greek small letter letter mu (U+03BC) instead of itself.
toLower :: Text -> Text Source
O(n) Convert a string to lower case, using simple case conversion. Subject to fusion.
The result string may be longer than the input string. For instance, the Latin capital letter I with dot above (U+0130) maps to the sequence Latin small letter i (U+0069) followed by combining dot above (U+0307).
toUpper :: Text -> Text Source
O(n) Convert a string to upper case, using simple case conversion. Subject to fusion.
The result string may be longer than the input string. For instance, the German eszett (U+00DF) maps to the two-letter sequence SS.
toTitle :: Text -> Text Source
O(n) Convert a string to title case, using simple case conversion. Subject to fusion.
The first letter of the input is converted to title case, as is every subsequent letter that immediately follows a non-letter. Every letter that immediately follows another letter is converted to lower case.
The result string may be longer than the input string. For example, the Latin small ligature fl (U+FB02) is converted to the sequence Latin capital letter F (U+0046) followed by Latin small letter l (U+006C).
Note: this function does not take language or culture specific rules into account. For instance, in English, different style guides disagree on whether the book name "The Hill of the Red Fox" is correctly title cased—but this function will capitalize every word.
Justification
justifyLeft :: Int64 -> Char -> Text -> Text Source
O(n) Left-justify a string to the given length, using the specified fill character on the right. Subject to fusion. Performs replacement on invalid scalar values.
Examples:
justifyLeft 7 'x' "foo" == "fooxxxx" justifyLeft 3 'x' "foobar" == "foobar"
justifyRight :: Int64 -> Char -> Text -> Text Source
O(n) Right-justify a string to the given length, using the specified fill character on the left. Performs replacement on invalid scalar values.
Examples:
justifyRight 7 'x' "bar" == "xxxxbar" justifyRight 3 'x' "foobar" == "foobar"
center :: Int64 -> Char -> Text -> Text Source
O(n) Center a string to the given length, using the specified fill character on either side. Performs replacement on invalid scalar values.
Examples:
center 8 'x' "HS" = "xxxHSxxx"
Folds
foldl' :: (a -> Char -> a) -> a -> Text -> a Source
O(n) A strict version of foldl. Subject to fusion.
foldl1' :: (Char -> Char -> Char) -> Text -> Char Source
O(n) A strict version of foldl1. Subject to fusion.
Special folds
Construction
Scans
Accumulating maps
mapAccumR :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text) Source
The mapAccumR function behaves like a combination of map and a strict foldr; it applies a function to each element of a Text, passing an accumulating parameter from right to left, and returning a final value of this accumulator together with the new Text. Performs replacement on invalid scalar values.
Generation and unfolding
unfoldr :: (a -> Maybe (Char, a)) -> a -> Text Source
O(n), where n is the length of the result. The unfoldr function is analogous to the List unfoldr. unfoldr builds a Text from a seed value. The function takes the element and returns Nothing if it is done producing the Text, otherwise Just (a,b). In this case, a is the next Char in the string, and b is the seed value for further production. Performs replacement on invalid scalar values.
unfoldrN :: Int64 -> (a -> Maybe (Char, a)) -> a -> Text Source
O(n) Like unfoldr, unfoldrN builds a Text from a seed value. However, the length of the result should be limited by the first argument to unfoldrN. This function is more efficient than unfoldr when the maximum length of the result is known and correct, otherwise its performance is similar to unfoldr. Performs replacement on invalid scalar values.
Substrings
Breaking strings
takeEnd :: Int64 -> Text -> Text Source
O(n) takeEnd n t returns the suffix remaining after taking n characters from the end of t.
Examples:
takeEnd 3 "foobar" == "bar"
dropEnd :: Int64 -> Text -> Text Source
O(n) dropEnd n t returns the prefix remaining after dropping n characters from the end of t.
Examples:
dropEnd 3 "foobar" == "foo"
dropWhileEnd :: (Char -> Bool) -> Text -> Text Source
O(n) dropWhileEnd p t returns the prefix remaining after dropping characters that fail the predicate p from the end of t. Examples:
dropWhileEnd (=='.') "foo..." == "foo"
dropAround :: (Char -> Bool) -> Text -> Text Source
O(n) dropAround p t returns the substring remaining after dropping characters that fail the predicate p from both the beginning and end of t. Subject to fusion.
O(n) Remove leading and trailing white space from a string. Equivalent to:
dropAround isSpace
stripStart :: Text -> Text Source
O(n) Remove leading white space from a string. Equivalent to:
dropWhile isSpace
stripEnd :: Text -> Text Source
O(n) Remove trailing white space from a string. Equivalent to:
dropWhileEnd isSpace
span :: (Char -> Bool) -> Text -> (Text, Text) Source
O(n) span, applied to a predicate p and text t, returns a pair whose first element is the longest prefix (possibly empty) of t of elements that satisfy p, and whose second is the remainder of the list.
breakOn :: Text -> Text -> (Text, Text) Source
O(n+m) Find the first instance of needle (which must be non-null) in haystack. The first element of the returned tuple is the prefix of haystack before needle is matched. The second is the remainder of haystack, starting with the match.
Examples:
breakOn "::" "a::b::c" ==> ("a", "::b::c") breakOn "/" "foobar" ==> ("foobar", "")Laws:
append prefix match == haystack where (prefix, match) = breakOn needle haystack
If you need to break a string by a substring repeatedly (e.g. you want to break on every instance of a substring), use breakOnAll instead, as it has lower startup overhead.
This function is strict in its first argument, and lazy in its second.
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
breakOnEnd :: Text -> Text -> (Text, Text) Source
O(n+m) Similar to breakOn, but searches from the end of the string.
The first element of the returned tuple is the prefix of haystack up to and including the last match of needle. The second is the remainder of haystack, following the match.
breakOnEnd "::" "a::b::c" ==> ("a::b::", "c")group :: Text -> [Text] Source
The group function takes a Text and returns a list of Texts such that the concatenation of the result is equal to the argument. Moreover, each sublist in the result contains only equal elements. For example,
group "Mississippi" = ["M","i","ss","i","ss","i","pp","i"]
It is a special case of groupBy, which allows the programmer to supply their own equality test.
Breaking into many substrings
Splitting functions in this library do not perform character-wise copies to create substrings; they just construct new Texts that are slices of the original.
Arguments
| :: Text | String to split on. If this string is empty, an error will occur. |
| -> Text | Input text. |
| -> [Text] |
O(m+n) Break a Text into pieces separated by the first Text argument (which cannot be an empty string), consuming the delimiter. An empty delimiter is invalid, and will cause an error to be raised.
Examples:
splitOn "\r\n" "a\r\nb\r\nd\r\ne" == ["a","b","d","e"] splitOn "aaa" "aaaXaaaXaaaXaaa" == ["","X","X","X",""] splitOn "x" "x" == ["",""]
and
intercalate s . splitOn s == id splitOn (singleton c) == split (==c)
(Note: the string s to split on above cannot be empty.)
This function is strict in its first argument, and lazy in its second.
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
split :: (Char -> Bool) -> Text -> [Text] Source
O(n) Splits a Text into components delimited by separators, where the predicate returns True for a separator element. The resulting components do not contain the separators. Two adjacent separators result in an empty component in the output. eg.
split (=='a') "aabbaca" == ["","","bb","c",""] split (=='a') [] == [""]
chunksOf :: Int64 -> Text -> [Text] Source
O(n) Splits a Text into components of length k. The last element may be shorter than the other chunks, depending on the length of the input. Examples:
chunksOf 3 "foobarbaz" == ["foo","bar","baz"] chunksOf 4 "haskell.org" == ["hask","ell.","org"]
Breaking into lines and words
Predicates
isPrefixOf :: Text -> Text -> Bool Source
O(n) The isPrefixOf function takes two Texts and returns True iff the first is a prefix of the second. Subject to fusion.
isSuffixOf :: Text -> Text -> Bool Source
O(n) The isSuffixOf function takes two Texts and returns True iff the first is a suffix of the second.
View patterns
stripPrefix :: Text -> Text -> Maybe Text Source
O(n) Return the suffix of the second string if its prefix matches the entire first string.
Examples:
stripPrefix "foo" "foobar" == Just "bar" stripPrefix "" "baz" == Just "baz" stripPrefix "foo" "quux" == Nothing
This is particularly useful with the ViewPatterns extension to GHC, as follows:
{-# LANGUAGE ViewPatterns #-} import Data.Text.Lazy as T fnordLength :: Text -> Int fnordLength (stripPrefix "fnord" -> Just suf) = T.length suf fnordLength _ = -1stripSuffix :: Text -> Text -> Maybe Text Source
O(n) Return the prefix of the second string if its suffix matches the entire first string.
Examples:
stripSuffix "bar" "foobar" == Just "foo" stripSuffix "" "baz" == Just "baz" stripSuffix "foo" "quux" == Nothing
This is particularly useful with the ViewPatterns extension to GHC, as follows:
{-# LANGUAGE ViewPatterns #-} import Data.Text.Lazy as T quuxLength :: Text -> Int quuxLength (stripSuffix "quux" -> Just pre) = T.length pre quuxLength _ = -1commonPrefixes :: Text -> Text -> Maybe (Text, Text, Text) Source
O(n) Find the longest non-empty common prefix of two strings and return it, along with the suffixes of each string at which they no longer match.
If the strings do not have a common prefix or either one is empty, this function returns Nothing.
Examples:
commonPrefixes "foobar" "fooquux" == Just ("foo","bar","quux") commonPrefixes "veeble" "fetzer" == Nothing commonPrefixes "" "baz" == NothingSearching
O(n+m) Find all non-overlapping instances of needle in haystack. Each element of the returned list consists of a pair:
- The entire string prior to the kth match (i.e. the prefix)
- The kth match, followed by the remainder of the string
Examples:
breakOnAll "::" "" ==> [] breakOnAll "/" "a/b/c/" ==> [("a", "/b/c/"), ("a/b", "/c/"), ("a/b/c", "/")]This function is strict in its first argument, and lazy in its second.
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
The needle parameter may not be empty.