External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-17 17:56:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
285 Commits 1 Branch 17 Tags
Commit Graph

132 Commits

Author SHA1 Message Date
Paul Wise
ba4ca60377
Do not emit a comma for the final item in JSON output 
This avoids producing JSON that is not parsable.
 2021-05-03 20:54:29 +08:00
Paul Wise
06e25957b6
Print progress messages to stderr when printing JSON 
This avoids the messages breaking JSON parsing when
the output is being redirected to a file and parsed.
 2021-05-03 20:52:28 +08:00
Paul Wise
cd29f79fd0
Switch to the JSON output format for easier parsing 2021-05-03 17:44:56 +08:00
DessertArbiter
15edae6a92 updated deprecated calls, changed URI to https 2020-05-27 20:28:06 -04:00
hartator
0a2ae60378 Bump Gem version 2017-10-26 20:09:23 -05:00
hartator
c360d4621f Merge branch 'master' of https://github.com/niklasjansson/wayback-machine-downloader into niklasjansson-master 2017-10-26 20:05:10 -05:00
hartator
aa5977c53a Bump Gem version 2017-10-26 19:44:00 -05:00
hartator
c06ab067aa Fix missing comma issue 2017-10-26 19:43:48 -05:00
hartator
a7c3d9b6c1 Merge branch 'pr/89' 
# Conflicts:
#	lib/wayback_machine_downloader.rb
 2017-10-26 19:39:36 -05:00
hartator
d80b51f502 Use more explicit variable name all_timestamps 2017-10-26 19:35:29 -05:00
hartator
3179faca2e Bump Gem version 2017-06-11 22:18:07 -05:00
hartator
123c2f3024 Bump Gem version 2017-06-11 22:06:57 -05:00
hartator
246441ff17 Replace exact match by exact url 2017-06-11 21:53:13 -05:00
hartator
4eca581257 Remove too verbose comment 2017-06-11 13:19:50 -05:00
hartator
28fd1e10a2 Fix length of arguments per line 2017-06-11 13:19:36 -05:00
hartator
62f424b6d1 Merge branch 'master' of https://github.com/p/wayback-machine-downloader into p-master 2017-06-11 12:46:00 -05:00
hartator
f465c83a80 Bump Gem version 2017-06-10 15:49:35 -05:00
Niklas Jansson
662ab9eeb7 fixed bad characters in directories for windows 2017-06-05 12:32:37 +02:00
Oleg Pudeyev
4af80adca6 Fix exact match logic and add a test 2017-06-03 17:45:06 -04:00
Oleg Pudeyev
e73a88ab56 File.exists? causes warning is ruby 2.4.1, use exist? 2017-06-03 16:53:37 -04:00
Oleg Pudeyev
d926f965f9 Add exact_match option. 
With this option set, Wayback Machine Downloader
will only look for snapshots matching the exact base_url
passed in rather than base_url and its children.

This is useful when trying to download a single file
rather than mirroring a site.
 2017-03-15 17:58:05 -04:00
Oleg Pudeyev
65b1948517 Avoid interleaving status output with file listing. 
Before:

[
Getting snapshot pages.. found 1 snaphots to consider.

{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]

After:

Getting snapshot pages.. found 1 snaphots to consider.

[
{"file_url":"http://www.trackpedia.com:80/forums/archive/index.php/f-115.html","timestamp":20131221124252,"file_id":"forums/archive/index.php/f-115.html"},
]
 2017-03-15 17:19:34 -04:00
Oleg Pudeyev
6b8c1aa194 Remove list attribute from the downloader. 
Whether to list or download is a program option external to the downloader
 2017-03-15 17:12:43 -04:00
Oleg Pudeyev
ea73ed5ed6 Shorten some lines for readability 2017-03-15 17:10:16 -04:00
hartator
e95ade8079 Bump Gem version 2017-02-17 12:55:58 -06:00
hartator
4830913ed3 Add explicit variable current encoding 2017-02-17 12:54:12 -06:00
Ian Kirker
132e3fa5f8 Alters encoding of file_url to fix encoding incompatibilities 2017-02-15 11:01:26 +00:00
insaner
5bd9fbffdd Update wayback_machine_downloader.rb 2017-01-24 05:03:07 -05:00
hartator
734521803a Bump Gem version 2016-11-14 18:20:16 -06:00
hartator
7383bdfadc Bump Gem version 2016-10-17 12:42:05 -05:00
hartator
9fc60b5385 Require thread library explicitly for 1.9.x Ruby versions 2016-10-17 12:42:05 -05:00
hartator
c9be8ae945 Bump Gem version 2016-09-26 10:36:05 -07:00
hartator
c9e15636b1 Fix spelling in comment 2016-09-26 10:36:05 -07:00
hartator
c1d77259c9 Bump Gem version 2016-09-24 10:21:50 -07:00
hartator
7414c34d05 Add an additional request to archive.org API to get fresher snapshots 2016-09-24 10:21:17 -07:00
hartator
188b285848 Bump gem version 2016-09-24 10:06:35 -07:00
hartator
1b42f57e6f Add maximum snapshot page option 2016-09-24 10:06:27 -07:00
hartator
f7afca7d95 Bump Gem version 2016-09-18 12:24:29 -05:00
hartator
2e7f8611ef Load early net/http library for Ruby 1.9.x 2016-09-17 14:08:29 -05:00
hartator
95eaa91715 Refactor archive API calls to own module 2016-09-17 13:37:17 -05:00
hartator
8e6dab2bba Bump Gem version 2016-09-17 12:49:56 -05:00
hartator
bd462ab36d Add possibility to save files to an user specified directory 2016-09-17 12:49:48 -05:00
hartator
8623d3a74b Bump Gem version 2016-09-16 19:17:53 -05:00
hartator
d4bc975043 Skip lines that don't include an url 2016-09-16 19:16:47 -05:00
hartator
bdf611bce9 Set file id to nil when encounter UTF-8 issues 2016-09-16 15:52:18 -05:00
hartator
94acdd6a7d Skip lines with no actionable data 2016-09-16 15:02:55 -05:00
hartator
50ee9e60be Replace String#split processing by substring selectors 2016-09-16 12:32:31 -05:00
hartator
7b69f3c236 Clean up data earlier from UTF-8 bad bytes 2016-09-16 12:09:58 -05:00
hartator
cfa236a4d1 Replace UTF-8 probematic blank space with another text editor 2016-09-16 12:04:47 -05:00
hartator
7da624d2bf Bump Gem version 2016-09-16 11:52:49 -05:00