1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
|
[[!toc levels=2]]
Mediawiki is a dynamically-generated wiki which stores it's data in a
relational database. Pages are marked up using a proprietary markup. It is
possible to import the contents of a Mediawiki site into an ikiwiki,
converting some of the Mediawiki conventions into Ikiwiki ones.
The following instructions describe ways of obtaining the current version of
the wiki. We do not yet cover importing the history of edits.
Another set of instructions and conversion tools (which imports the full history)
can be found at <http://github.com/mithro/media2iki>
## Step 1: Getting a list of pages
The first bit of information you require is a list of pages in the Mediawiki.
There are several different ways of obtaining these.
### Parsing the output of `Special:Allpages`
Mediawikis have a special page called `Special:Allpages` which list all the
pages for a given namespace on the wiki.
If you fetch the output of this page to a local file with something like
wget -q -O tmpfile 'http://your-mediawiki/wiki/Special:Allpages'
You can extract the list of page names using the following python script. Note
that this script is sensitive to the specific markup used on the page, so if
you have tweaked your mediawiki theme a lot from the original, you will need
to adjust this script too:
import sys
from xml.dom.minidom import parse, parseString
dom = parse(sys.argv[1])
tables = dom.getElementsByTagName("table")
pagetable = tables[-1]
anchors = pagetable.getElementsByTagName("a")
for a in anchors:
print a.firstChild.toxml().\
replace('&','&').\
replace('<','<').\
replace('>','>')
Also, if you have pages with titles that need to be encoded to be represented
in HTML, you may need to add further processing to the last line.
Note that by default, `Special:Allpages` will only list pages in the main
namespace. You need to add a `&namespace=XX` argument to get pages in a
different namespace. (See below for the default list of namespaces)
Note that the page names obtained this way will not include any namespace
specific prefix: e.g. `Category:` will be stripped off.
### Querying the database
If you have access to the relational database in which your mediawiki data is
stored, it is possible to derive a list of page names from this. With mediawiki's
MySQL backend, the page table is, appropriately enough, called `table`:
SELECT page_namespace, page_title FROM page;
As with the previous method, you will need to do some filtering based on the
namespace.
### namespaces
The list of default namespaces in mediawiki is available from <http://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces>. Here are reproduced the ones you are most likely to encounter if you are running a small mediawiki install for your own purposes:
[[!table data="""
Index | Name | Example
0 | Main | Foo
1 | Talk | Talk:Foo
2 | User | User:Jon
3 | User talk | User_talk:Jon
6 | File | File:Barack_Obama_signature.svg
10 | Template | Template:Prettytable
14 | Category | Category:Pages_needing_review
"""]]
## Step 2: fetching the page data
Once you have a list of page names, you can fetch the data for each page.
### Method 1: via HTTP and `action=raw`
You need to create two derived strings from the page titles: the
destination path for the page and the source URL. Assuming `$pagename`
contains a pagename obtained above, and `$wiki` contains the URL to your
mediawiki's `index.php` file:
src=`echo "$pagename" | tr ' ' _ | sed 's,&,&,g'`
dest=`"$pagename" | tr ' ' _ | sed 's,&,__38__,g'`
mkdir -p `dirname "$dest"`
wget -q "$wiki?title=$src&action=raw" -O "$dest"
You may need to add more conversions here depending on the precise page titles
used in your wiki.
If you are trying to fetch pages from a different namespace to the default,
you will need to prefix the page title with the relevant prefix, e.g.
`Category:` for category pages. You probably don't want to prefix it to the
output page, but you may want to vary the destination path (i.e. insert an
extra directory component corresponding to your ikiwiki's `tagbase`).
### Method 2: via HTTP and `Special:Export`
Mediawiki also has a special page `Special:Export` which can be used to obtain
the source of the page and other metadata such as the last contributor, or the
full history, etc.
You need to send a `POST` request to the `Special:Export` page. See the source
of the page fetched via `GET` to determine the correct arguments.
You will then need to write an XML parser to extract the data you need from
the result.
### Method 3: via the database
It is possible to extract the page data from the database with some
well-crafted queries.
## Step 3: format conversion
The next step is to convert Mediawiki conventions into Ikiwiki ones.
### categories
Mediawiki uses a special page name prefix to define "Categories", which
otherwise behave like ikiwiki tags. You can convert every Mediawiki category
into an ikiwiki tag name using a script such as
import sys, re
pattern = r'\[\[Category:([^\]]+)\]\]'
def manglecat(mo):
return '\[[!tag %s]]' % mo.group(1).strip().replace(' ','_')
for line in sys.stdin.readlines():
res = re.match(pattern, line)
if res:
sys.stdout.write(re.sub(pattern, manglecat, line))
else: sys.stdout.write(line)
## Step 4: Mediawiki plugin
The [[plugins/contrib/mediawiki]] plugin can be used by ikiwiki to interpret
most of the Mediawiki syntax.
The following things are not working:
* templates
* tables
* spaces and other funky characters ("?") in page names
## Scripts
[[sabr]] used to explain how to [import MediaWiki content into
git](http://u32.net/Mediawiki_Conversion/index.html?updated), including full
edit history, but as of 2009/10/16 that site is not available. A copy of the
information found on this website is stored at <http://github.com/mithro/media2iki>.
[[Albert]] wrote a ruby script to convert from mediawiki's database to ikiwiki at <https://github.com/docunext/mediawiki2gitikiwiki>
[[Anarcat]] wrote a python script to convert from a mediawiki website to ikiwiki at <http://anarcat.ath.cx/software/mediawikigitdump.git/>. The script doesn't need any special access or privileges and communicates with the documented API (so it's a bit slower, but allows you to mirror sites you are not managing, like parts of Wikipedia). The script can also incrementally import new changes from a running site, through RecentChanges inspection. It also supports mithro's new Mediawiki2markdown converter.
> Some assembly is required to get Mediawiki2markdown and its mwlib
> gitmodule available in the right place for it to use.. perhaps you could
> automate that? --[[Joey]]
> > You mean a debian package? :) media2iki is actually a submodule, so you need to go through extra steps to install it. mwlib being the most annoying part... I have fixed my script so it looks for media2iki directly in the submodule and improved the install instructions in the README file, but I'm not sure I can do much more short of starting to package the whole thing... --[[anarcat]]
>>> You may have forgotten to push that, I don't see those changes.
>>> Packaging the python library might be a good 1st step.
>>> --[[Joey]]
> Also, when I try to run it with -t on www.amateur-radio-wiki.net, it
> fails on some html in the page named "4_metres". On archiveteam.org,
> it fails trying to write to a page filename starting with "/", --[[Joey]]
> > can you show me exactly which commandline arguments you're using? also, I have made improvements over the converter too, also available here: <http://anarcat.ath.cx/software/media2iki.git/> -- [[anarcat]]
>>> Not using your new converter, just the installation I did earlier
>>> today:
>>> --[[Joey]]
<pre>
fetching page 4 metres from http://www.amateur-radio-wiki.net//index.php?action=raw&title=4+metres into 4_metres.mdwn
Unknown tag TagNode tagname='div' vlist={'style': {u'float': u'left', u'border': u'2px solid #aaa', u'margin-left': u'20px'}}->'div' div
Traceback (most recent call last):
File "./mediawikigitdump.py", line 298, in <module>
fetch_allpages(namespace)
File "./mediawikigitdump.py", line 82, in fetch_allpages
fetch_page(page.getAttribute('title'))
File "./mediawikigitdump.py", line 187, in fetch_page
c.parse(urllib.urlopen(url).read())
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 285, in parse
self.parse_node(ast)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 76, in parse_node
f(node)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 88, in on_article
self.parse_children(node)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 83, in parse_children
self.parse_node(child)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 76, in parse_node
f(node)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 413, in on_section
self.parse_node(child)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 76, in parse_node
f(node)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 83, in parse_children
self.parse_node(child)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 76, in parse_node
f(node)
File "/home/joey/tmp/mediawikigitdump/mediawiki2markdown.py", line 474, in on_tagnode
assert not options.STRICT
AssertionError
zsh: exit 1 ./mediawikigitdump.py -v -t http://www.amateur-radio-wiki.net/
</pre>
<pre>
joey@wren:~/tmp/mediawikigitdump>./mediawikigitdump.py -v -t http://archiveteam.org
fetching page list from namespace 0 ()
found 222 pages
fetching page /Sites using MediaWiki (English) from http://archiveteam.org/index.php?action=raw&title=%2FSites+using+MediaWiki+%28English%29 into /Sites_using_MediaWiki_(English).mdwn
Traceback (most recent call last):
File "./mediawikigitdump.py", line 298, in <module>
fetch_allpages(namespace)
File "./mediawikigitdump.py", line 82, in fetch_allpages
fetch_page(page.getAttribute('title'))
File "./mediawikigitdump.py", line 188, in fetch_page
f = open(filename, 'w')
IOError: [Errno 13] Permission denied: u'/Sites_using_MediaWiki_(English).mdwn'
zsh: exit 1 ./mediawikigitdump.py -v -t http://archiveteam.org
</pre>
[[scy]] wrote a python script to convert from mediawiki XML dumps to git repositories at <https://github.com/scy/levitation>.
|