A place to cache linked articles (think custom and personal wayback machine)
您最多选择25个主题 主题必须以字母或数字开头,可以包含连字符 (-),并且长度不得超过35个字符

3 年前
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485
  1. title: Host your own wikipedia backup
  2. url: https://dataswamp.org/~solene/2019-11-13-wikimedia-dump.html
  3. hash_url: 626841dd876f103a10391b1b841ba5ae
  4. <h2 id="wikipediaandopenzim">Wikipedia and openzim</h2>
  5. <p>If you ever wanted to host your own wikipedia replica, here is the simplest
  6. way.</p>
  7. <p>As wikipedia is REALLY huge, you don’t really want to host a php wikimedia
  8. software and load the huge database, instead, the project made the <em>openzim</em>
  9. format to compress the huge database that wikipedia became while allowing using
  10. it for fast searches.</p>
  11. <p>Sadly, on OpenBSD, we have no software reading zim files and most software
  12. requires the library openzim to work which requires extra work to get it as a
  13. package on OpenBSD.</p>
  14. <p>Hopefully, there is a python package implementing all you need as pure python
  15. to serve zim files over http and it’s easy to install.</p>
  16. <p>This tutorial should work on all others unix like systems but packages or
  17. binary names may change.</p>
  18. <h2 id="downloadingwikipedia">Downloading wikipedia</h2>
  19. <p>The project Kiwix is responsible for wikipedia files, they create regularly
  20. files from various projects (including stackexchange, gutenberg, wikibooks
  21. etc…) but for this tutorial we want wikipedia:
  22. <a href="https://wiki.kiwix.org/wiki/Content_in_all_languages">https://wiki.kiwix.org/wiki/Content_in_all_languages</a></p>
  23. <p>You will find a lot of files, the language is contained into the filename. Some
  24. filenames will also self explain if they contain everything or categories, and
  25. if they have pictures or not.</p>
  26. <p>The full French file is 31.4 GB worth.</p>
  27. <h2 id="runningtheserver">Running the server</h2>
  28. <p>For the next steps, I recommend setting up a new user dedicated to this.</p>
  29. <p>On OpenBSD, we will require python3 and pip:</p>
  30. <pre><code>$ doas pkg_add py3-pip--
  31. </code></pre>
  32. <p>Then we can use pip to fetch and install dependencies for the zimply software,
  33. the flag <code>--user</code> is rather important as it allows any user to download and
  34. install python libraries in its home folder instead of polluting the whole
  35. system as root.</p>
  36. <pre><code>$ pip3.7 install --user --upgrade zimply
  37. </code></pre>
  38. <p>I wrote a small script to start the server using the zim file as a parameter, I
  39. rarely write python so the script may not be high standard.</p>
  40. <p>File <strong>server.py</strong>:</p>
  41. <pre><code>from zimply import ZIMServer
  42. import sys
  43. import os.path
  44. if len(sys.argv) == 1:
  45. print("usage: " + sys.argv[0] + " file")
  46. exit(1)
  47. if os.path.exists(sys.argv[1]):
  48. ZIMServer(sys.argv[1])
  49. else:
  50. print("Can't find file " + sys.argv[1])
  51. </code></pre>
  52. <p>And then you can start the server using the command:</p>
  53. <pre><code>$ python3.7 server.py /path/to/wikipedia_fr_all_maxi_2019-08.zim
  54. </code></pre>
  55. <p>You will be able to access wikipedia on the url http://localhost:9454/</p>
  56. <p>Note that this is not a “wiki” as you can’t see history and edit/create pages.</p>
  57. <p>This kind of backup is used in place like Cuba or Africa areas where people
  58. don’t have unlimited internet access, the project lead by Kiwix allow more
  59. people to access knowledge.</p>