External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-29 17:16:06 +01:00
Code Issues Packages Projects Releases Wiki Activity
446 Commits 1 Branch 17 Tags
master
Commit Graph
224 Commits
Author SHA1 Message Date
FelipeandGitHub d572207122 Improve URL detection for timestamps + fix retry download links 2025-12-18 14:41:35 +00:00
FelipeandGitHub b2fc748c2c another page_requisites fix 2025-12-10 12:16:26 +00:00
FelipeandGitHub 8632050c45 page requisites fix 2025-12-10 12:13:39 +00:00
FelipeandGitHub 2aa694eed0 Initial implementation of --page-requisites 
see StrawberryMaster/wayback-machine-downloader#39
 2025-12-10 11:59:00 +00:00
FelipeandGitHub 4d2513eca8 Be a bit more tolerant of timeouts here 2025-11-15 12:59:07 +00:00
FelipeandGitHub 67685b781e Improve handling for wildcard URLs 
fixes #38
 2025-11-15 12:45:34 +00:00
FelipeandGitHub f7c0f1a964 Better support for .php, .asp, and other files when using --local 
see #37
 2025-11-04 23:18:04 +00:00
FelipeandGitHub 71bdc7c2de Use explicit current directory to avoid ambiguity 
see `Results saved in /build/websites` but nothing is saved :(
Fixes StrawberryMaster/wayback-machine-downloader#34
 2025-10-27 16:48:15 +00:00
FelipeandGitHub d7a63361e3 Use a FixedThreadPool for concurrent API calls 2025-09-24 21:05:22 +00:00
FelipeandGitHub b1974a8dfa Refactor ConnectionPool to use SizedQueue for connection management and improve cleanup logic 2025-09-24 20:50:10 +00:00
adampwebandGitHub dec9083b43 Fix: Fixed trivial mistake with function call 2025-09-04 19:24:44 +00:00
FelipeandGitHub c517bd20d3 Actual retry implementation 
seems I pushed an older revision of this apparently
 2025-09-04 19:16:52 +00:00
FelipeandGitHub fc8d8a9441 Added retry command 
fixes [Feature request} Retry flag
Fixes StrawberryMaster/wayback-machine-downloader#31
 2025-08-20 01:21:29 +00:00
FelipeandGitHub fa306ac92b Bumped version 2025-08-19 16:17:53 +00:00
FelipeandGitHub 8c27aaebc9 Fix issue with index.html pages not loading 
we were rejecting empty paths, causing these files to be skipped. How did I miss this?
 2025-08-19 16:16:24 +00:00
FelipeandGitHub 40e9c9bb51 Bumped version 2025-08-16 19:38:01 +00:00
Felipe 6bc08947b7 More aggressive sanitization 
this should deal with some of the issues we've seen, luckily. What a ride!
 2025-08-12 18:55:00 -03:00
FelipeandGitHub c731e0c7bd Bumped version 2025-08-12 11:46:03 +00:00
Felipe 9fd2a7f8d1 Minor refactoring of HTML tag sanitization 2025-08-12 08:42:27 -03:00
FelipeandGitHub 6ad312f31f Sanitizing HTML tags 
some sites contain tags *in* their URL, and fail to save on some devices like Windows
 2025-08-05 23:44:34 +00:00
FelipeandGitHub 62ea35daa6 Bumping version 2025-08-04 21:23:48 +00:00
Felipe 1f4202908f Fixes for tidy_bytes 
admittedly not the cleanest way to do this, although it works for #25.
 2025-07-31 12:58:22 -03:00
adampweb 801fb77f79 Perf: Refactored a huge function into smaller subprocesses 2025-07-29 21:12:20 +02:00
FelipeandGitHub bc868e6b39 Refactor tidy_bytes.rb 
I'm not sure if we can easily determine the encoding behind each site (and I don't think Wayback Machine does that), *but* we can at least translate it and get it to download. This should be mostly useful for other, non-Western European languages. See #25
 2025-07-29 10:10:56 -03:00
FelipeandGitHub 2bf04aff48 Sanitize base_url and directory parameters 
this might be the cause of #25, at least from what it appears
 2025-07-27 17:18:57 +00:00
FelipeandGitHub 51becde916 Minor fix 2025-07-26 21:01:40 +00:00
FelipeandGitHub c30ee73977 Sanitize file_id 
we were not consistently handling non-UTF-8 characters here, especially after commit e4487baafc. This also fixes #25
 2025-07-26 20:58:50 +00:00
Felipe d3466b3387 Bumping version 
normally I would've yanked the old gem, but that's not working here
 2025-07-22 12:41:26 +00:00
FelipeandGitHub 0663c1c122 Merge pull request #23 from adampweb/master 
Fixed base image vulnerability
 2025-07-21 14:44:43 -03:00
Felipe bff10e7260 Initial implementation of a composite snapshot 
see issue #22. TBF
 2025-07-21 15:30:49 +00:00
Felipe 3d181ce84c Bumped version 2025-07-21 13:48:34 +00:00
Alfonso CorradoandFelipe 999aa211ae fix match filters 2025-07-21 13:42:44 +00:00
adampweb e4487baafc Fix: Handle default case in tidy_bytes 2025-07-20 17:13:36 +02:00
FelipeandGitHub fd329afdd2 Merge pull request #20 from underarchiver/rfc3968-url-validity-check 
Prevent fetching off non RFC3968-compliant URLs
 2025-07-11 10:55:12 -03:00
FelipeandGitHub 038785557d Ability to recursively download across subdomains 
this is quite experimental. Fixes #15 but still needs more testing
 2025-07-09 12:53:58 +00:00
FelipeandGitHub 2eead8cc27 Bumping version 2025-06-27 19:50:39 +00:00
cybercode3andGitHub 7e5cdd54fb Fix: path sanitizer and timestamp sorting errors 
Fix: path sanitizer and timestamp sorting errors

( I encountered these errors issues with the script using Windows 11. Changing these two lines got the script to work for me. )

- Fixed a bug in Windows path sanitizer where String#gsub was incorrectly called with a Proc as the replacement. Replaced with block form to ensure proper character escaping for Windows-incompatible file path characters.
- Fixed an ArgumentError in file sorting when a file snapshot’s timestamp was nil. Updated sort logic to safely handle nil timestamps by converting them to strings or integers, preventing comparison errors between NilClass and String/Integer.

These changes prevent fatal runtime errors when downloading files with certain URLs or incomplete metadata, improving robustness for sites with inconsistent archive data.
 2025-06-25 02:07:20 +00:00
FelipeandGitHub 4160ff5e4a Bumping version 2025-06-18 18:05:31 +00:00
underarchiver f03d92a3c4 Prevent fetching off non RFC3968-compliant URLs 2025-06-17 13:27:10 +02:00
Eli Dickinson c3c5b8446a don’t append /* when —exact-url 2025-06-15 13:26:11 -04:00
FelipeandGitHub 18357a77ed Correct file path and sanitization in Windows 
Not only we weren't normalizing the file directories, we were also agressively sanitizing incorrect characters, leading to some funny stuff on Windows. Fixes #16
 2025-06-15 13:48:11 +00:00
FelipeandGitHub 3fdfd70fc1 Bump version 2025-06-05 22:34:40 +00:00
Eli Dickinson 79cbb639e7 Fix bug with archive urls containing square brackets 2025-06-03 16:36:03 -04:00
Eli Dickinson 1681a12579 workaround for API only showing html files for some domains 
See https://github.com/StrawberryMaster/wayback-machine-downloader/issues/6
 2025-05-30 12:50:48 -04:00
FelipeandGitHub f38756dd76 Correction for downloaded data folder 
if you downloaded content from example.org/*, it would be listed in a folder titled * instead of the sitename. See #6 (and thanks to elidickinson for pointing it out!)
 2025-05-30 14:00:32 +00:00
FelipeandGitHub 9452411e32 Added nil checks 2025-05-30 13:52:25 +00:00
FelipeandGitHub 61e22cfe25 Bump versions 2025-05-27 18:10:09 +00:00
FelipeandGitHub 183ed61104 Attempt at fixing --all 
I honestly don't recall if this was implemented in the original code, and I'm guessing this worked at *some point* during this fork. It seems to work correctly now, however. See #6 and #11
 2025-05-27 17:17:34 +00:00
FelipeandGitHub ab4324c0eb Bumping to 2.3.6 2025-05-18 16:49:44 +00:00
FelipeandGitHub e28d7d578b Experimental ability to rewrite URLs to local browsing 2025-05-18 16:48:50 +00:00