External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-17 17:56:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
259 Commits 1 Branch 17 Tags
Commit Graph

119 Commits

Author SHA1 Message Date
hartator
123c2f3024 Bump Gem version 2017-06-11 22:06:57 -05:00
hartator
246441ff17 Replace exact match by exact url 2017-06-11 21:53:13 -05:00
hartator
4eca581257 Remove too verbose comment 2017-06-11 13:19:50 -05:00
hartator
28fd1e10a2 Fix length of arguments per line 2017-06-11 13:19:36 -05:00
hartator
62f424b6d1 Merge branch 'master' of https://github.com/p/wayback-machine-downloader into p-master 2017-06-11 12:46:00 -05:00
hartator
f465c83a80 Bump Gem version 2017-06-10 15:49:35 -05:00
Oleg Pudeyev
4af80adca6 Fix exact match logic and add a test 2017-06-03 17:45:06 -04:00
Oleg Pudeyev
e73a88ab56 File.exists? causes warning is ruby 2.4.1, use exist? 2017-06-03 16:53:37 -04:00
Oleg Pudeyev
d926f965f9 Add exact_match option. 
With this option set, Wayback Machine Downloader
will only look for snapshots matching the exact base_url
passed in rather than base_url and its children.

This is useful when trying to download a single file
rather than mirroring a site.
 2017-03-15 17:58:05 -04:00
Oleg Pudeyev
65b1948517 Avoid interleaving status output with file listing. 
Before:

[
Getting snapshot pages.. found 1 snaphots to consider.

{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]

After:

Getting snapshot pages.. found 1 snaphots to consider.

[
{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]
 2017-03-15 17:19:34 -04:00
Oleg Pudeyev
6b8c1aa194 Remove list attribute from the downloader. 
Whether to list or download is a program option external to the downloader
 2017-03-15 17:12:43 -04:00
Oleg Pudeyev
ea73ed5ed6 Shorten some lines for readability 2017-03-15 17:10:16 -04:00
hartator
e95ade8079 Bump Gem version 2017-02-17 12:55:58 -06:00
hartator
4830913ed3 Add explicit variable current encoding 2017-02-17 12:54:12 -06:00
Ian Kirker
132e3fa5f8 Alters encoding of file_url to fix encoding incompatibilities 2017-02-15 11:01:26 +00:00
hartator
734521803a Bump Gem version 2016-11-14 18:20:16 -06:00
hartator
7383bdfadc Bump Gem version 2016-10-17 12:42:05 -05:00
hartator
9fc60b5385 Require thread library explicitly for 1.9.x Ruby versions 2016-10-17 12:42:05 -05:00
hartator
c9be8ae945 Bump Gem version 2016-09-26 10:36:05 -07:00
hartator
c9e15636b1 Fix spelling in comment 2016-09-26 10:36:05 -07:00
hartator
c1d77259c9 Bump Gem version 2016-09-24 10:21:50 -07:00
hartator
7414c34d05 Add an additional request to archive.org API to get fresher snapshots 2016-09-24 10:21:17 -07:00
hartator
188b285848 Bump gem version 2016-09-24 10:06:35 -07:00
hartator
1b42f57e6f Add maximum snapshot page option 2016-09-24 10:06:27 -07:00
hartator
f7afca7d95 Bump Gem version 2016-09-18 12:24:29 -05:00
hartator
2e7f8611ef Load early net/http library for Ruby 1.9.x 2016-09-17 14:08:29 -05:00
hartator
95eaa91715 Refactor archive API calls to own module 2016-09-17 13:37:17 -05:00
hartator
8e6dab2bba Bump Gem version 2016-09-17 12:49:56 -05:00
hartator
bd462ab36d Add possibility to save files to an user specified directory 2016-09-17 12:49:48 -05:00
hartator
8623d3a74b Bump Gem version 2016-09-16 19:17:53 -05:00
hartator
d4bc975043 Skip lines that don't include an url 2016-09-16 19:16:47 -05:00
hartator
bdf611bce9 Set file id to nil when encounter UTF-8 issues 2016-09-16 15:52:18 -05:00
hartator
94acdd6a7d Skip lines with no actionable data 2016-09-16 15:02:55 -05:00
hartator
50ee9e60be Replace String#split processing by substring selectors 2016-09-16 12:32:31 -05:00
hartator
7b69f3c236 Clean up data earlier from UTF-8 bad bytes 2016-09-16 12:09:58 -05:00
hartator
cfa236a4d1 Replace UTF-8 probematic blank space with another text editor 2016-09-16 12:04:47 -05:00
hartator
7da624d2bf Bump Gem version 2016-09-16 11:52:49 -05:00
hartator
e508d761c8 Fix UTF-8 blank space issues with Ruby 1.9.x and JRuby 2016-09-16 11:52:15 -05:00
hartator
dd9c91b433 Bump Gem version 2016-09-15 20:15:50 -05:00
hartator
87c28b0895 Fix random UTF-8 issues with Ruby 1.9 and JRuby 2016-09-15 20:15:40 -05:00
hartator
fbfa8f8e63 Bump Gem version 2016-09-15 20:02:36 -05:00
hartator
5696853e23 Format seconds to avoid too long output 2016-09-15 20:00:35 -05:00
hartator
85a9fae671 Make instance variable definitions more explicit 2016-09-15 19:59:42 -05:00
Šarūnas Kūjalis
ee5c87378d Add downloading in multi threads 2016-09-04 23:38:38 +03:00
hartator
82a938b628 Add support for raw domain names as inputed url 2016-08-16 11:48:00 -05:00
hartator
d6f65250cd Bump Gem version 2016-08-16 11:48:00 -05:00
hartator
53f621e6a1 Bump Gem version 2016-08-10 14:33:21 -05:00
hartator
e7fc5a2eb9 Fix case when file wasn't created 2016-08-10 14:32:41 -05:00
hartator
5532a6408b Bump gem version 2016-08-09 20:56:54 -05:00
hartator
762ccb20c7 Filter more windows unsafe characters 2016-08-09 20:56:47 -05:00