{-# OPTIONS_GHC -fno-warn-unused-imports #-} {-| @pipes-parse@ builds upon @pipes@ to add several missing features necessary to implement 'Parser's: * End-of-input detection, so that 'Parser's can react to an exhausted input stream * Leftovers support, which simplifies several parsing problems * Connect-and-resume, to connect a 'Producer' to a 'Parser' and retrieve unused input -} module Pipes.Parse.Tutorial ( -- * Overview -- $overview -- * Parsers -- $parsers -- * Lenses -- $lenses -- * Getters -- $getters -- * Building Lenses -- $buildlenses -- * Conclusion -- $conclusion ) where import Pipes import Pipes.Parse {- $overview @pipes-parse@ centers on three abstractions: * 'Producer's, unchanged from @pipes@ * 'Parser's, which play a role analogous to 'Consumer's * 'Lens.Family2.Lens''es between 'Producer's, which play a role analogous to 'Pipe's There are four ways to connect these three abstractions: * Connect 'Parser's to 'Producer's using 'runStateT' \/ 'evalStateT' \/ 'execStateT': > runStateT :: Parser a m r -> Producer a m x -> m (r, Producer a m x) > evalStateT :: Parser a m r -> Producer a m x -> m r > execStateT :: Parser a m r -> Producer a m x -> m ( Producer a m x) * Connect 'Lens.Family2.Lens''es to 'Parser's using 'Lens.Family.State.Strict.zoom' > zoom :: Lens' (Producer a m x) (Producer b m y) > -> Parser b m r > -> Parser a m r * Connect 'Producer's to 'Lens.Family2.Lens''es using ('Lens.Family.^.') or 'Lens.Family.view': > (^.) :: Producer a m x > -> Lens' (Producer a m x) (Producer b m y) > -> Producer b m y * Connect 'Lens.Family2.Lens''es to 'Lens.Family2.Lens''es using ('.') (i.e. function composition): > (.) :: Lens' (Producer a m x) (Producer b m y) > -> Lens' (Producer b m y) (Producer c m z) > -> Lens' (Producer a m x) (Producer c m z) You can obtain the necessary lens utilities from either: * The @lens-family-core@ library, importing @Lens.Family@ (for ('Lens.Family.^.') \/ 'Lens.Family.view' and 'Lens.Family.over') and @Lens.Family.State.Strict@ (for 'Lens.Family.State.Strict.zoom'), or: * The @lens@ library, importing @Control.Lens@ (for ('Control.Lens.^.') \/ 'Control.Lens.view', 'Control.Lens.over' and 'Control.Lens.zoom') This tutorial uses @Lens.Family@ since it has fewer dependencies and simpler types. -} {- $parsers 'Parser's handle end-of-input and pushback by storing a 'Producer' in a 'StateT' layer: > type Parser a m r = forall x . StateT (Producer a m x) m r To draw a single element from the underlying 'Producer', use the 'draw' command: > draw :: Monad m => Parser a m (Maybe a) 'draw' returns the next element from the 'Producer' wrapped in 'Just' or returns 'Nothing' if the underlying 'Producer' is empty. Here's an example 'Parser' written using 'draw' that retrieves the first two elements from a stream: > import Pipes.Parse > > drawTwo :: Monad m => Parser a m (Maybe a, Maybe a) > drawTwo = do > mx <- draw > my <- draw > return (mx, my) > > -- or: drawTwo = liftM2 (,) draw draw Since a 'Parser' is just a 'StateT' action, you run a 'Parser' using the same run functions as 'StateT': > -- Feed a 'Producer' to a 'Parser', returning the result and leftovers > runStateT :: Parser a m r -> Producer a m x -> m (r, Producer a m x) > > -- Feed a 'Producer' to a 'Parser', returning only the result > evalStateT :: Parser a m r -> Producer a m x -> m r > > -- Feed a 'Producer' to a 'Parser', returning only the leftovers > execStateT :: Parser a m r -> Producer a m x -> m ( Producer a m x) All three of these functions require a 'Producer' which we feed to the 'Parser'. For example, we can feed standard input: >>> evalStateT drawTwo Pipes.Prelude.stdinLn Pink<Enter> Elephants<Enter> (Just "Pink",Just "Elephants") The result is wrapped in a 'Maybe' because 'draw' can fail if the 'Producer' is empty: >>> evalStateT drawTwo (yield 0) (Just 0,Nothing) Parsing might not necessarily consume the entire stream. We can use 'runStateT' or 'execStateT' to retrieve unused elements that our parser does not consume: >>> import Pipes >>> (result, unused) <- runStateT drawTwo (each [1..4]) >>> -- View the parsed result >>> result (Just 1,Just 2) >>> -- Now print the leftovers >>> runEffect $ for unused (lift . print) 3 4 -} {- $lenses @pipes-parse@ also provides a convenience function for testing purposes that draws all remaining elements and returns them as a list: > drawAll :: Monad m => Parser a m [a] For example: >>> import Pipes >>> import Pipes.Parse >>> evalStateT drawAll (each [1..10]) [1,2,3,4,5,6,7,8,9,10] However, this function is not recommended in general because it loads the entire input into memory, which defeats the purpose of streaming parsing. You can instead use 'foldAll' if you wish to fold all input elements into a single result: >>> evalStateT (foldAll (+) 0 id) (each [1..10]) 55 You can also use the @foldl@ package to simplify writing more complex folds: >>> import Control.Applicative >>> import Control.Foldl as L >>> evalStateT (purely foldAll (liftA2 (,) L.sum L.maximum)) (each [1..10]) (55,Just 10) But what if you wanted to draw or fold just the first three elements from an infinite stream instead of the entire input? This is what lenses are for: > import Lens.Family > import Lens.Family.State.Strict > import Pipes > import Pipes.Parse > > import Prelude hiding (splitAt, span) > > drawThree :: Monad m => Parser a m [a] > drawThree = zoom (splitAt 3) drawAll 'Lens.Family.State.Strict.zoom' lets you delimit a 'Parser' using a 'Lens.Family2.Lens''. The above code says to limit 'drawAll' to a subset of the input, in this case the first three elements: >>> evalStateT drawThree (each [1..]) [1,2,3] 'splitAt' is a 'Lens.Family2.Lens'' with the following type: > splitAt > :: Monad m > => Int -> Lens' (Producer a m x) (Producer a m (Producer a m x)) The easiest way to understand 'splitAt' is to study what happens when you use it as a getter: > view (splitAt 3) :: Producer a m x -> Producer a m (Producer a m x) In this context, @(splitAt 3)@ behaves like 'splitAt' from the Prelude, except instead of splitting a list it splits a 'Producer'. Here's an example of how you can use 'splitAt': > outer :: Monad m => Producer Int m (Producer Int m ()) > outer = each [1..6] ^. splitAt 3 The above definition of @outer@ is exactly equivalent to: > outer = do > each [1..3] > return (each [4..6]) We can prove this by successively running the outer and inner 'Producer' layers: >>> -- Print all the elements in the outer layer and return the inner layer >>> inner <- runEffect $ for outer (lift . print) 1 2 3 >>> -- Now print the elements in the inner layer >>> runEffect $ for inner (lift . print) 4 5 6 We can also uses lenses to modify 'Parser's, using 'Lens.Family.State.Strict.zoom'. When we combine 'Lens.Family.State.Strict.zoom' with @(splitAt 3)@ we limit a parser to the the first three elements of the stream. When the parser is done 'Lens.Family.State.Strict.zoom' also returns unused elements back to the original stream. We can demonstrate this using the following example parser: > splitExample :: Monad m => Parser a m ([a], Maybe a, [a]) > splitExample = do > x <- zoom (splitAt 3) drawAll > y <- zoom (splitAt 3) draw > z <- zoom (splitAt 3) drawAll > return (x, y, z) The second parser begins where the first parser left off: >>> evalStateT splitExample (each [1..]) ([1,2,3],Just 4,[5,6,7]) 'span' behaves the same way, except that it uses a predicate and takes as many consecutive elements as possible that satisfy the predicate: > spanExample :: Monad m => Parser Int m (Maybe Int, [Int], Maybe Int) > spanExample = do > x <- zoom (span (>= 4)) draw > y <- zoom (span (< 4)) drawAll > z <- zoom (span (>= 4)) draw > return (x, y, z) Note that even if the first parser fails, subsequent parsers can still succeed because they operate under a different lens: >>> evalStateT spanExample (each [1..]) (Nothing,[1,2,3],Just 4) You can even nest 'Lens.Family.State.Strict.zoom's, too: > nestExample :: Monad m => Parser Int m (Maybe Int, [Int], Maybe Int) > nestExample = zoom (splitAt 2) spanExample All the parsers from @spanExample@ now only see a subset of the input, namely the first two elements: >>> evalStateT nestExample (each [1..]) (Nothing,[1,2],Nothing) -} {- $getters Not all transformations are reversible. For example, consider the following contrived function: > import Pipes > import qualified Pipes.Prelude as P > > map' :: Monad m => (a -> b) -> Producer a m r -> Producer b m r > map' f p = p >-> P.map f Given a function of type @(a -> b)@, we can transform a stream of @a@'s into a stream of @b@'s, but not the other way around. Transformations which are not reversible and cannot be modeled as 'Pipe's can only be modeled as functions between 'Producer's. However, 'Pipe's are preferable to functions between 'Producer's when possible because 'Pipe's can transform both 'Producer's and 'Consumer's. If you prefer, you can use lens-like syntax for functions between 'Producer's by promoting them to @Getter@s using 'Lens.Family.to': > import Lens.Family > > example :: Monad m => Producer Int m () > example = each [1..3] ^. to (map' (*2)) However, a function of 'Producer's (or the equivalent @Getter@) cannot be used transform 'Parser's (using 'Lens.Family.State.Strict.zoom' or otherwise) . This reflects the fact that such a transformation cannot be applied in reversed. -} {- $buildlenses Lenses are very easy to write if you are willing to depend on either the @lens-family@ or @lens@ library. Both of these libraries provide an 'Lens.Family2.Unchecked.iso' function that you can use to assemble your own lenses. You only need two functions which reversibly transform back and forth between a stream of @a@s and a stream of @b@s: > -- "Forward" > fw :: Producer a m x -> Producer b m y > > -- "Backward" > bw :: Producer b m y -> Producer a m x ... such that: > fw . bw = id > > bw . fw = id You can then convert them to a 'Lens.Family2.Lens'' using 'Lens.Family2.Unchecked.iso': > import Lens.Family2 (Lens') > import Lens.Family2.Unchecked (iso) > > lens :: Lens' (Producer a m x) (Producer b m y) > lens = iso fw bw You can even do this without incurring any dependencies if you rewrite the above code like this: > -- This type synonym requires the 'RankNTypes' extension > type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) > > lens :: Lens' (Producer a m x) (Producer b m y) > lens k p = fmap bw (k (fw p)) This is what @pipes-parse@ does internally, and you will find several examples of this pattern in the source code of the "Pipes.Parse" module. Lenses defined using either approach will work with both the @lens@ and @lens-family@ libraries. -} {- $conclusion @pipes-parse@ introduces core idioms for @pipes@-based parsing. These idioms reuse 'Producer's, but introduce two new abstractions: 'Lens.Family2.Lens''es and 'Parser's. This library is very minimal and only contains datatype-agnostic parsing utilities, so this tutorial does not explore the full range of parsing tricks using lenses. For example, you can also use lenses to change the element type. Several downstream libraries provide more specific functionality, including: * @pipes-binary@: Lenses and parsers for @binary@ values * @pipes-attoparsec@: Converts @attoparsec@ parsers to @pipes@ parsers * @pipes-aeson@: Lenses and parsers for JSON values * @pipes-bytestring@: Lenses and parsers for byte streams * @pipes-text@: Lenses and parsers for text encodings To learn more about @pipes-parse@, ask questions, or follow development, you can subscribe to the @haskell-pipes@ mailing list at: <https://groups.google.com/forum/#!forum/haskell-pipes> ... or you can mail the list directly at: <mailto:haskell-pipes@googlegroups.com> -}