From 7b24072546fdcda160ac37f4abc6d6e0bc6c8f81 Mon Sep 17 00:00:00 2001 From: "http://www.cse.unsw.edu.au/~willu/" Date: Wed, 20 May 2009 21:30:16 -0400 Subject: Responses --- doc/todo/tracking_bugs_with_dependencies.mdwn | 32 ++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/doc/todo/tracking_bugs_with_dependencies.mdwn b/doc/todo/tracking_bugs_with_dependencies.mdwn index 8b36f1e59..a3250f178 100644 --- a/doc/todo/tracking_bugs_with_dependencies.mdwn +++ b/doc/todo/tracking_bugs_with_dependencies.mdwn @@ -196,21 +196,39 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W > Very belated code review of last version of the patch: > > * `is_globlist` is no longer needed + +>> Good :) + > * I don't understand why the pagespec match regexp is changed > from having flags `igx` to `ixgs`. Don't see why you > want `.` to match '\n` in it, and don't see any `.` in the regexp > anyway? + +>> Because you have to define all the named pagespecs in the pagespec, you sometimes end up with very long pagespecs. I found it useful to split them over multiple lines. That didn't work at one point and I added the 's' to make it work. I may have further altered the regex since then to make the 's' redundant. Remove it and see if multi-line pagespecs still work. :) + > * Some changes of `@_` to `%params` in `pagespec_makeperl` do not > make sense to me. I don't see where \%params is defined and populated, > except with `\$params{specFunc}`. + +>> I'm not a perl hacker. This was a mighty battle for me to get going. There is probably some battlefield carnage from my early struggles learning perl left here. +>> Part of this is that @_ / @params already existed as a way of passing in extra parameters. I didn't want to pollute that top level namespace - just at my own parameter (a hash) which contained the data I needed. + > * Seems that the only reason `match_glob` has to check for `~` is > because when a named spec appears in a pagespec, it is translated > to `match_glob("~foo")`. If, instead, `pagespec_makeperl` checked > for named specs, it could convert them into `check_named_spec("foo")` > and avoid that ugliness. + +>> Yeah - I wanted to make named specs syntactically different on my first pass. You are right in that this could be made a fallback - named specs always override pagenames. + > * The changes to `match_link` seem either unecessary, or incomplete. > Shouldn't it check for named specs and call > `check_named_spec_existential`? + +>> An earlier version did. Then I realised it wasn't actually needed in that case - match_link() already included a loop that was like a type of existential matching. Each time through the loop it would +>> call match_glob(). match_glob() in turn will handle the named spec. I tested this version briefly and it seemed to work. I remember looking at this again later and wondering if I had mis-understood +>> some of the logic in match_link(), which might mean there are cases where you would need an explicit call to check_named_spec_existential() - I never checked it properly after having that thought. + > * Generally, the need to modify `match_*` functions so that they > check for and handle named pagespecs seems suboptimal, if > only because there might be others people may want to use named @@ -221,13 +239,25 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W > that is not a page name at all, and it could be weird > if such a parameter were accidentially interpreted as a named > pagespec. (But, that seems unlikely to happen.) + +>> Possibly. I'm not sure which I prefer between the current solution and that one. Each have advantages and disadvantages. +>> It really isn't much code for the match functions to add a call to check_named_spec_existential(). + > * I need to check if your trick to avoid infinite recursion > works if there are two named specs that recursively > call one-another. I suspect it does, but will test this > myself.. -> + +>> It worked for me. :) + > --[[Joey]] +>> There is one issue that I've been thinking about that I haven't raised anywhere (or checked myself), and that is how this all interacts with page dependencies. +>> Firstly, I'm not sure anymore that the `pagespec_merge` function will continue to work in all cases. Secondly, it seems that there are two types of dependency, and ikiwiki +>> currently only handles one of them. The first type is "Rebuild this page when any of these other pages changes" - ikiwiki handles this. The second type is "rebuild this page when +>> set of pages referred to by this pagespec changes" - ikiwiki doesn't seem to handle this. I suspect that named pagespecs would make that second type of dependency more +>> important. I'll try to come up with a good example. -- [[Will]] + ---- diff --git a/IkiWiki.pm b/IkiWiki.pm -- cgit v1.2.3 From 6a0cffc41b0cac86d444138b05140f2043c96c80 Mon Sep 17 00:00:00 2001 From: "http://www.cse.unsw.edu.au/~willu/" Date: Thu, 21 May 2009 00:56:53 -0400 Subject: Comment on potential bug --- doc/todo/tracking_bugs_with_dependencies.mdwn | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/todo/tracking_bugs_with_dependencies.mdwn b/doc/todo/tracking_bugs_with_dependencies.mdwn index a3250f178..464f68363 100644 --- a/doc/todo/tracking_bugs_with_dependencies.mdwn +++ b/doc/todo/tracking_bugs_with_dependencies.mdwn @@ -258,6 +258,9 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W >> set of pages referred to by this pagespec changes" - ikiwiki doesn't seem to handle this. I suspect that named pagespecs would make that second type of dependency more >> important. I'll try to come up with a good example. -- [[Will]] +>>> Hrm, I was going to build an example of this with backlinks, but it looks like that is handled as a special case at the moment (line 458 of render.pm). I'll see if I can break +>>> things another way. Fixing this properly would allow removal of that special case. -- [[Will]] + ---- diff --git a/IkiWiki.pm b/IkiWiki.pm -- cgit v1.2.3 From 7188942440a434fcdc5943cb5cc5c15d9a4d65e7 Mon Sep 17 00:00:00 2001 From: "http://jmtd.net/" Date: Thu, 21 May 2009 05:15:11 -0400 Subject: non tag pages under the tagbase --- doc/bugs/tagged__40____41___matching_wikilinks.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn index 1bd556f50..40b5dd9e9 100644 --- a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn +++ b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn @@ -15,3 +15,5 @@ this is due to the wikilink being equal to a `\[[!tag ...]]`. What's the rationale on this, or what am I doing wrong, and how to achieve what I want? --[[tschwinge]] + +: What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. -- [[Jon]] -- cgit v1.2.3 From af7f2271ba8e35fbc49bd6b18fd9b8cf67ddc60e Mon Sep 17 00:00:00 2001 From: "http://jmtd.net/" Date: Thu, 21 May 2009 05:15:55 -0400 Subject: syntax confusion: tried to use ':' to indent. --- doc/bugs/tagged__40____41___matching_wikilinks.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn index 40b5dd9e9..5de59fe88 100644 --- a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn +++ b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn @@ -16,4 +16,4 @@ rationale on this, or what am I doing wrong, and how to achieve what I want? --[[tschwinge]] -: What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. -- [[Jon]] +> What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. -- [[Jon]] -- cgit v1.2.3 From d2071292f0f56954f01f52c0c8fd02922e9d5c21 Mon Sep 17 00:00:00 2001 From: "http://jmtd.net/" Date: Thu, 21 May 2009 05:17:08 -0400 Subject: --- doc/bugs/tagged__40____41___matching_wikilinks.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn index 5de59fe88..bdd0c93c7 100644 --- a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn +++ b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn @@ -16,4 +16,4 @@ rationale on this, or what am I doing wrong, and how to achieve what I want? --[[tschwinge]] -> What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. -- [[Jon]] +> What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. The rationale for implementing tag as it has been, I think, is one of simplicity and conciseness. -- [[Jon]] -- cgit v1.2.3 From bcf3f9091a945467a7a68b5b828915e65f0ea139 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 21 May 2009 12:44:21 -0400 Subject: add test case for multiline pagespec --- t/pagespec_match.t | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/t/pagespec_match.t b/t/pagespec_match.t index 4cf6fa1ff..64408f489 100755 --- a/t/pagespec_match.t +++ b/t/pagespec_match.t @@ -1,7 +1,7 @@ #!/usr/bin/perl use warnings; use strict; -use Test::More tests => 53; +use Test::More tests => 54; BEGIN { use_ok("IkiWiki"); } @@ -20,6 +20,15 @@ ok(! pagespec_match("foo", "* and !foo")); ok(! pagespec_match("foo", "foo and !foo")); ok(! pagespec_match("foo.png", "* and !*.*")); ok(pagespec_match("foo", "(bar or ((meep and foo) or (baz or foo) or beep))")); +ok(pagespec_match("foo", "( + bar + or ( + (meep and foo) + or + (baz or foo) + or beep + ) +)"), "multiline complex pagespec"); ok(! pagespec_match("a/foo", "foo", location => "a/b"), "nonrelative fail"); ok(! pagespec_match("foo", "./*", location => "a/b"), "relative fail"); ok(pagespec_match("a/foo", "./*", location => "a/b"), "relative"); -- cgit v1.2.3 From f97f102b04e7ae67bd3db5320dd34496309de5fe Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 21 May 2009 13:33:46 -0400 Subject: response --- doc/todo/tracking_bugs_with_dependencies.mdwn | 83 ++++++++++++++++++++++++--- 1 file changed, 75 insertions(+), 8 deletions(-) diff --git a/doc/todo/tracking_bugs_with_dependencies.mdwn b/doc/todo/tracking_bugs_with_dependencies.mdwn index 464f68363..417d5910e 100644 --- a/doc/todo/tracking_bugs_with_dependencies.mdwn +++ b/doc/todo/tracking_bugs_with_dependencies.mdwn @@ -206,12 +206,51 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W >> Because you have to define all the named pagespecs in the pagespec, you sometimes end up with very long pagespecs. I found it useful to split them over multiple lines. That didn't work at one point and I added the 's' to make it work. I may have further altered the regex since then to make the 's' redundant. Remove it and see if multi-line pagespecs still work. :) +>>> Well, I can tell you that multi-line pagespecs are supported w/o +>>> your patch .. I use them all the time. The reason I find your +>>> use of `/s` unlikely is because without it `\s` already matches +>>> a newline. Only if you want to treat a newline as non-whitespace +>>> is `/s` typically necessary. --[[Joey]] + > * Some changes of `@_` to `%params` in `pagespec_makeperl` do not > make sense to me. I don't see where \%params is defined and populated, > except with `\$params{specFunc}`. ->> I'm not a perl hacker. This was a mighty battle for me to get going. There is probably some battlefield carnage from my early struggles learning perl left here. ->> Part of this is that @_ / @params already existed as a way of passing in extra parameters. I didn't want to pollute that top level namespace - just at my own parameter (a hash) which contained the data I needed. +>> I'm not a perl hacker. This was a mighty battle for me to get going. +>> There is probably some battlefield carnage from my early struggles +>> learning perl left here. Part of this is that @_ / @params already +>> existed as a way of passing in extra parameters. I didn't want to +>> pollute that top level namespace - just at my own parameter (a hash) +>> which contained the data I needed. + +>>> I think I understand how the various `%params` +>>> (there's not just one) work in your code now, but it's really a mess. +>>> Explaining it in words would take pages.. It could be fixed by, +>>> in `pagespec_makeperl` something like: +>>> +>>> my %specFuncs; +>>> push @_, specFuncs => \%specFuncs; +>>> +>>> With that you have the hash locally available for populating +>>> inside `pagespec_makeperl`, and when the `match_*` functions +>>> are called the same hash data will be available inside their +>>> `@_` or `%params`. No need to change how the functions are called +>>> or do any of the other hacks. +>>> +>>> Currently, specFuncs is populated by building up code +>>> that recursively calls `pagespec_makeperl`, and is then +>>> evaluated when the pagespec gets evaluated. My suggested +>>> change to `%params` will break that, but that had to change +>>> anyway. +>>> +>>> It probably has a security hole, and is certianly inviting +>>> one, since the pagespec definition is matched by a loose regexp (`.*`) +>>> and then subject to string interpolation before being evaluated +>>> inside perl code. I recently changed ikiwiki to never interpolate +>>> user-supplied strings when translating pagespecs, and that +>>> needs to happen here too. The obvious way, it seems to me, +>>> is to not generate perl code, but just directly run perl code that +>>> populates specFuncs. > * Seems that the only reason `match_glob` has to check for `~` is > because when a named spec appears in a pagespec, it is translated @@ -229,6 +268,10 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W >> call match_glob(). match_glob() in turn will handle the named spec. I tested this version briefly and it seemed to work. I remember looking at this again later and wondering if I had mis-understood >> some of the logic in match_link(), which might mean there are cases where you would need an explicit call to check_named_spec_existential() - I never checked it properly after having that thought. +>>> In the common case, `match_link` does not call `match_glob`, +>>> because the link target it is being asked to check for is a single +>>> page name, not a glob. + > * Generally, the need to modify `match_*` functions so that they > check for and handle named pagespecs seems suboptimal, if > only because there might be others people may want to use named @@ -243,6 +286,9 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W >> Possibly. I'm not sure which I prefer between the current solution and that one. Each have advantages and disadvantages. >> It really isn't much code for the match functions to add a call to check_named_spec_existential(). +>>> But if a plugin adds its own match function, it has +>>> to explicitly call that code to support named pagespecs. + > * I need to check if your trick to avoid infinite recursion > works if there are two named specs that recursively > call one-another. I suspect it does, but will test this @@ -250,17 +296,38 @@ account all comments above (which doesn't mean it is above reproach :) ). --[[W >> It worked for me. :) +> * I also need to verify if memoizing the named pagespecs has +> really guarded against very expensive pagespecs DOSing the wiki.. + > --[[Joey]] >> There is one issue that I've been thinking about that I haven't raised anywhere (or checked myself), and that is how this all interacts with page dependencies. ->> Firstly, I'm not sure anymore that the `pagespec_merge` function will continue to work in all cases. Secondly, it seems that there are two types of dependency, and ikiwiki ->> currently only handles one of them. The first type is "Rebuild this page when any of these other pages changes" - ikiwiki handles this. The second type is "rebuild this page when ->> set of pages referred to by this pagespec changes" - ikiwiki doesn't seem to handle this. I suspect that named pagespecs would make that second type of dependency more ->> important. I'll try to come up with a good example. -- [[Will]] - ->>> Hrm, I was going to build an example of this with backlinks, but it looks like that is handled as a special case at the moment (line 458 of render.pm). I'll see if I can break +>> Firstly, I'm not sure anymore that the `pagespec_merge` function will continue to work in all cases. + +>>> The problem I can see there is that if two pagespecs +>>> get merged and both use `~foo` but define it differently, +>>> then the second definition might be used at a point when +>>> it shouldn't (but I haven't verified that really happens). +>>> That could certianly be a show-stopper. --[[Joey]] + +>> Secondly, it seems that there are two types of dependency, and ikiwiki +>> currently only handles one of them. The first type is "Rebuild this +>> page when any of these other pages changes" - ikiwiki handles this. +>> The second type is "rebuild this page when set of pages referred to by +>> this pagespec changes" - ikiwiki doesn't seem to handle this. I +>> suspect that named pagespecs would make that second type of dependency +>> more important. I'll try to come up with a good example. -- [[Will]] + +>>> Hrm, I was going to build an example of this with backlinks, but it +>>> looks like that is handled as a special case at the moment (line 458 of +>>> render.pm). I'll see if I can breapk >>> things another way. Fixing this properly would allow removal of that special case. -- [[Will]] +>>>> I can't quite understand the distinction you're trying to draw +>>>> between the two types of dependencies. Backlinks are a very special +>>>> case though and I'll be suprised if they fit well into pagespecs. +>>>> --[[Joey]] + ---- diff --git a/IkiWiki.pm b/IkiWiki.pm -- cgit v1.2.3 From 49635b0ca563b336a5ffc1d17a4197ee8b437018 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 21 May 2009 13:54:02 -0400 Subject: response --- doc/bugs/tagged__40____41___matching_wikilinks.mdwn | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn index bdd0c93c7..2b30f247e 100644 --- a/doc/bugs/tagged__40____41___matching_wikilinks.mdwn +++ b/doc/bugs/tagged__40____41___matching_wikilinks.mdwn @@ -1,7 +1,10 @@ It may be that I'm simply misunderstanding something, but what is the rationale for having `tagged()` also match normal wikilinks? -> It simply hasn't been implemented yet -- see the answer in [[todo/tag_pagespec_function]]. Tags and wikilinks share the same underlying implementation, although ab reasonable expectation is that they are kept separate. --Ivan Z. +> It simply hasn't been implemented yet -- see the answer in +> [[todo/tag_pagespec_function]]. Tags and wikilinks share the same +> underlying implementation, although ab reasonable expectation is that +> they are kept separate. --Ivan Z. The following situation. I have `tagbase => 'tag'`. On some pages, scattered over the whole wiki, I use `\[[!tag open_issue_gdb]]` to declare that this page @@ -16,4 +19,11 @@ rationale on this, or what am I doing wrong, and how to achieve what I want? --[[tschwinge]] -> What you are doing "wrong" is putting non-tag pages (i.e. `/tag/open_issues_gdb.mdwn`) under your tagbase. The rationale for implementing tag as it has been, I think, is one of simplicity and conciseness. -- [[Jon]] +> What you are doing "wrong" is putting non-tag pages (i.e. +> `/tag/open_issues_gdb.mdwn`) under your tagbase. The rationale for +> implementing tag as it has been, I think, is one of simplicity and +> conciseness. -- [[Jon]] + +>> No, he has no pages under tagbase that aren't tags. This bug +>> is valid. [[todo/matching_different_kinds_of_links]] is probably +>> how it will eventually be solved. --[[Joey]] -- cgit v1.2.3 From 1a677f862c8a5e1e446adcea3f3737de07085c35 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 21 May 2009 14:00:55 -0400 Subject: response --- doc/bugs/SSI_include_stripped_from_mdwn.mdwn | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/doc/bugs/SSI_include_stripped_from_mdwn.mdwn b/doc/bugs/SSI_include_stripped_from_mdwn.mdwn index bd895127a..5519e45c6 100644 --- a/doc/bugs/SSI_include_stripped_from_mdwn.mdwn +++ b/doc/bugs/SSI_include_stripped_from_mdwn.mdwn @@ -1,3 +1,21 @@ If I have a <--#include virtual="foo" --> in some file, it gets stripped, even though other HTML comments don't get stripped. I imagine it's some plugin doing it, or IkiWiki itself, or an IkiWiki dependency, but I haven't found where this is happening. I'm trying to implement a workaround for my sidebars forcing a rebuild of the wiki every day - I use the calendar plugin - when the day changes, by using SSI. > It is probably the [[plugins/htmlscrubber]] plugin. -- [[Jon]] + +> htmlscrubber does strip these, because they look like +> a html tag to it, not a html comment. (html comments start +> with `