External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-17 17:56:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
423 Commits 1 Branch 17 Tags
Commit Graph

181 Commits

Author SHA1 Message Date
Paul Wise
06e25957b6
Print progress messages to stderr when printing JSON 
This avoids the messages breaking JSON parsing when
the output is being redirected to a file and parsed.
 2021-05-03 20:52:28 +08:00
Paul Wise
cd29f79fd0
Switch to the JSON output format for easier parsing 2021-05-03 17:44:56 +08:00
DessertArbiter
15edae6a92 updated deprecated calls, changed URI to https 2020-05-27 20:28:06 -04:00
hartator
0a2ae60378 Bump Gem version 2017-10-26 20:09:23 -05:00
hartator
c360d4621f Merge branch 'master' of https://github.com/niklasjansson/wayback-machine-downloader into niklasjansson-master 2017-10-26 20:05:10 -05:00
hartator
aa5977c53a Bump Gem version 2017-10-26 19:44:00 -05:00
hartator
c06ab067aa Fix missing comma issue 2017-10-26 19:43:48 -05:00
hartator
a7c3d9b6c1 Merge branch 'pr/89' 
# Conflicts:
#	lib/wayback_machine_downloader.rb
 2017-10-26 19:39:36 -05:00
hartator
d80b51f502 Use more explicit variable name all_timestamps 2017-10-26 19:35:29 -05:00
hartator
3179faca2e Bump Gem version 2017-06-11 22:18:07 -05:00
hartator
123c2f3024 Bump Gem version 2017-06-11 22:06:57 -05:00
hartator
246441ff17 Replace exact match by exact url 2017-06-11 21:53:13 -05:00
hartator
4eca581257 Remove too verbose comment 2017-06-11 13:19:50 -05:00
hartator
28fd1e10a2 Fix length of arguments per line 2017-06-11 13:19:36 -05:00
hartator
62f424b6d1 Merge branch 'master' of https://github.com/p/wayback-machine-downloader into p-master 2017-06-11 12:46:00 -05:00
hartator
f465c83a80 Bump Gem version 2017-06-10 15:49:35 -05:00
Niklas Jansson
662ab9eeb7 fixed bad characters in directories for windows 2017-06-05 12:32:37 +02:00
Oleg Pudeyev
4af80adca6 Fix exact match logic and add a test 2017-06-03 17:45:06 -04:00
Oleg Pudeyev
e73a88ab56 File.exists? causes warning is ruby 2.4.1, use exist? 2017-06-03 16:53:37 -04:00
Oleg Pudeyev
d926f965f9 Add exact_match option. 
With this option set, Wayback Machine Downloader
will only look for snapshots matching the exact base_url
passed in rather than base_url and its children.

This is useful when trying to download a single file
rather than mirroring a site.
 2017-03-15 17:58:05 -04:00
Oleg Pudeyev
65b1948517 Avoid interleaving status output with file listing. 
Before:

[
Getting snapshot pages.. found 1 snaphots to consider.

{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]

After:

Getting snapshot pages.. found 1 snaphots to consider.

[
{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]
 2017-03-15 17:19:34 -04:00
Oleg Pudeyev
6b8c1aa194 Remove list attribute from the downloader. 
Whether to list or download is a program option external to the downloader
 2017-03-15 17:12:43 -04:00
Oleg Pudeyev
ea73ed5ed6 Shorten some lines for readability 2017-03-15 17:10:16 -04:00
hartator
e95ade8079 Bump Gem version 2017-02-17 12:55:58 -06:00
hartator
4830913ed3 Add explicit variable current encoding 2017-02-17 12:54:12 -06:00
Ian Kirker
132e3fa5f8 Alters encoding of file_url to fix encoding incompatibilities 2017-02-15 11:01:26 +00:00
insaner
5bd9fbffdd Update wayback_machine_downloader.rb 2017-01-24 05:03:07 -05:00
hartator
734521803a Bump Gem version 2016-11-14 18:20:16 -06:00
hartator
7383bdfadc Bump Gem version 2016-10-17 12:42:05 -05:00
hartator
9fc60b5385 Require thread library explicitly for 1.9.x Ruby versions 2016-10-17 12:42:05 -05:00
hartator
c9be8ae945 Bump Gem version 2016-09-26 10:36:05 -07:00
hartator
c9e15636b1 Fix spelling in comment 2016-09-26 10:36:05 -07:00
hartator
c1d77259c9 Bump Gem version 2016-09-24 10:21:50 -07:00
hartator
7414c34d05 Add an additional request to archive.org API to get fresher snapshots 2016-09-24 10:21:17 -07:00
hartator
188b285848 Bump gem version 2016-09-24 10:06:35 -07:00
hartator
1b42f57e6f Add maximum snapshot page option 2016-09-24 10:06:27 -07:00
hartator
f7afca7d95 Bump Gem version 2016-09-18 12:24:29 -05:00
hartator
2e7f8611ef Load early net/http library for Ruby 1.9.x 2016-09-17 14:08:29 -05:00
hartator
95eaa91715 Refactor archive API calls to own module 2016-09-17 13:37:17 -05:00
hartator
8e6dab2bba Bump Gem version 2016-09-17 12:49:56 -05:00
hartator
bd462ab36d Add possibility to save files to an user specified directory 2016-09-17 12:49:48 -05:00
hartator
8623d3a74b Bump Gem version 2016-09-16 19:17:53 -05:00
hartator
d4bc975043 Skip lines that don't include an url 2016-09-16 19:16:47 -05:00
hartator
bdf611bce9 Set file id to nil when encounter UTF-8 issues 2016-09-16 15:52:18 -05:00
hartator
94acdd6a7d Skip lines with no actionable data 2016-09-16 15:02:55 -05:00
hartator
50ee9e60be Replace String#split processing by substring selectors 2016-09-16 12:32:31 -05:00
hartator
7b69f3c236 Clean up data earlier from UTF-8 bad bytes 2016-09-16 12:09:58 -05:00
hartator
cfa236a4d1 Replace UTF-8 probematic blank space with another text editor 2016-09-16 12:04:47 -05:00
hartator
7da624d2bf Bump Gem version 2016-09-16 11:52:49 -05:00
hartator
e508d761c8 Fix UTF-8 blank space issues with Ruby 1.9.x and JRuby 2016-09-16 11:52:15 -05:00