External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-17 17:56:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add an additional request to archive.org API to get fresher snapshots

Browse Source
This commit is contained in:
hartator 2016-09-24 10:21:17 -07:00
parent 45f5f29826
commit 7414c34d05
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
lib/wayback_machine_downloader.rb
View File

@ -77,10 +77,13 @@ class WaybackMachineDownloader
end end
def get_all_snapshots_to_consider def get_all_snapshots_to_consider
# Note: Passing a page index parameter allow us to get more snapshot, but from a less fresh index
print "Getting snapshot pages" print "Getting snapshot pages"
snapshot_list_to_consider = "" snapshot_list_to_consider = ""
snapshot_list_to_consider += get_raw_list_from_api(@base_url, nil) snapshot_list_to_consider += get_raw_list_from_api(@base_url, nil)
print "." print "."
snapshot_list_to_consider += get_raw_list_from_api(@base_url + '/*', nil)
print "."
@maximum_pages.times do |page_index| @maximum_pages.times do |page_index|
snapshot_list = get_raw_list_from_api(@base_url + '/*', page_index) snapshot_list = get_raw_list_from_api(@base_url + '/*', page_index)
break if snapshot_list.empty? break if snapshot_list.empty?