Make a ZIM of http://cyclowiki.org/wiki/

[kelson42]

From @kelson42 on August 26, 2018 14:15

Copied from original issue: openzim/mwoffliner#363

[kelson42]

mwoffliner --mwUrl="http://cyclowiki.org" --adminEmail="[email protected]" --localParsoid --verbose

dies with

Executing command : pngquant --verbose --strip --nofs --force --ext=".ag2w2.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.png" && advdef -q -z -4 -i 5 "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.ag2w2.png" && if [ $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.ag2w2.png") -lt $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.png") ]; then mv "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.ag2w2.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.png"; else rm "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.ag2w2.png"; fi
Executing command : pngquant --verbose --strip --nofs --force --ext=".2db4l.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.png" && advdef -q -z -4 -i 5 "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.2db4l.png" && if [ $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.2db4l.png") -lt $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.png") ]; then mv "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.2db4l.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.png"; else rm "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.2db4l.png"; fi
Executing command : pngquant --verbose --strip --nofs --force --ext=".b7rl0.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.png" && advdef -q -z -4 -i 5 "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.b7rl0.png" && if [ $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.b7rl0.png") -lt $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.png") ]; then mv "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.b7rl0.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.png"; else rm "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.b7rl0.png"; fi
Executing command : pngquant --verbose --strip --nofs --force --ext=".a5s3f.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.png" && advdef -q -z -4 -i 5 "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.a5s3f.png" && if [ $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.a5s3f.png") -lt $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.png") ]; then mv "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.a5s3f.png" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.png"; else rm "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.a5s3f.png"; fi
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/video-icon.png
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/audio-icon.png
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/document-icon.png
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icons.png
Saving favicon.png...
Downloading http://cyclowiki.org/w/api.php?action=query&meta=siteinfo&format=json...
Executing command : gifsicle --verbose --colors 64 -O3 "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.gif" -o "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.tz4ir.gif" && if [ $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.tz4ir.gif") -lt $(stat -c%s "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.gif") ]; then mv "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.tz4ir.gif" "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.gif"; else rm "/srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.tz4ir.gif"; fi
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/user-icon.png
Successfuly optimized /srv/kiwix-maintenance/mwoffliner/tmp/cyclowiki_ru_all_2018-09/s/watch-icon-loading.gif
TypeError [ERR_INVALID_ARG_TYPE]: The "url" argument must be of type string. Received type undefined
    at Url.parse (url.js:150:11)
    at Object.urlParse [as parse] (url.js:144:13)
    at /usr/local/lib/node_modules/mwoffliner/lib/mwoffliner.lib.js:2083:29
    at async.retry (/usr/local/lib/node_modules/mwoffliner/lib/Downloader.js:176:7)
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:676:51
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:726:13
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:52:16
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:264:21
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:44:16
    at /usr/local/lib/node_modules/mwoffliner/node_modules/async/lib/async.js:723:17

[kelson42]

From @ISNIT0 on September 18, 2018 9:39

No logoUrl is returned by the api for this site (http://cyclowiki.org/w/api.php?action=query&meta=siteinfo&format=json)

What should be done in this situation?
Use a placeholder?

[kelson42]

@ISNIT0 It should stop with a proper error msg inviting to use --customZimFavicon

[kelson42]

Impacted by openzim/mwoffliner#387

[kelson42]

Done at https://farm.openzim.org/recipes/cyclowiki

[benoit74]

Still not published

[benoit74]

Recipe moved to https://farm.openzim.org/recipes/cyclowiki.org_rus_all

[benoit74]

We are blocked by Cloudflare, no matter which User-Agent / Referer we pass, Cloudflare still blocks us on first call to sanitize_mwUrl (i.e. probably first call ever) on most workers.

[Popolechien]

So close as wontfix or do we wait for a decision on mwoffliner#2134?

[benoit74]

#2134 might help, so probably worth to wait