[darcs-users] darcs replace token complaint

[Eric Kow]

On Fri, Aug 14, 2009 at 11:35:08 +1000, Trent W. Buck wrote:
> Neither of the two (sigh) default tokenizers allow / within a token.

Ah, so for reference, here they are:

default_toks :: String
default_toks = "A-Za-z_0-9"
filename_toks :: String
filename_toks = "A-Za-z_0-9\\-\\."

So the answer to Gwern would be that he needs to include '/' in his
token spec if he wants this work.

> I really do think that the "darcs replace" in its current form is too
> dangerous to use, because absolutely nothing has the exact lexical
> structure assumed by it.

The word "dangerous" made me sit up for a moment.  Then I realised you
were using a broader definition of dangerous than I am.  My stricter
sense of 'danger' here means 'uh-oh we have to change the Darcs core'
which does not appear to be the case here.

Are you at least convinced that darcs replace is safe in the sense that
it is simple/predictable?  Here is the code that implements the heart of
it.  There appears to be very little fancy machinery in there, just
using a (Char -> Bool) function (see RegChars) to determine if something
is a token character or a delimiter.

forceTokReplace :: String -> String -> String
                -> FileContents -> Maybe FileContents
forceTokReplace t os ns c = Just $ unlinesPS $ map forceReplace $ linesPS c
    where o = BC.pack os
          n = BC.pack ns
          tokchar = regChars t
          toks_and_intratoks ps | B.null ps = []
          toks_and_intratoks ps =
              let (before,s') = BC.break tokchar ps
                  (tok, after) = BC.break (not . tokchar) s'
                  in before : tok : toks_and_intratoks after
          forceReplace ps = B.concat $ map o_t_n $ toks_and_intratoks ps
          o_t_n s | s == o = n
                  | otherwise = s

Is that convincing enough?  Because if we only have to change the
default token spec in the replace command, that's easy.

Basically, I'm "happy" if once we have a replace patch it does exactly
what it says it will do 100% predictably.

No cause for Erician alarm, right?

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20090815/8bed8e70/attachment.pgp>