External/wayback-machine-downloader
1
0
Fork 0
mirror of https://github.com/StrawberryMaster/wayback-machine-downloader.git synced 2025-12-29 16:16:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • d572207122 Improve URL detection for timestamps + fix retry download links master Felipe 2025-12-18 14:41:35 +00:00
  • b2fc748c2c another page_requisites fix Felipe 2025-12-10 12:16:26 +00:00
  • 8632050c45 page requisites fix Felipe 2025-12-10 12:13:39 +00:00
  • 2aa694eed0 Initial implementation of --page-requisites see StrawberryMaster/wayback-machine-downloader#39 Felipe 2025-12-10 11:57:24 +00:00
  • 4d2513eca8 Be a bit more tolerant of timeouts here Felipe 2025-11-15 12:59:07 +00:00
  • 67685b781e Improve handling for wildcard URLs fixes #38 Felipe 2025-11-15 12:45:34 +00:00
  • f7c0f1a964 Better support for .php, .asp, and other files when using --local see #37 Felipe 2025-11-04 23:18:04 +00:00
  • 99da3ca48e Fix Docker command volume mount path in README (#35) Nicolai Weitkemper 2025-10-28 19:30:19 +01:00
  • 34f22c128c Bump to 2.4.4 v2.4.4 Felipe 2025-10-27 16:51:58 +00:00
  • 71bdc7c2de Use explicit current directory to avoid ambiguity see Results saved in /build/websites but nothing is saved :( Fixes StrawberryMaster/wayback-machine-downloader#34 Felipe 2025-10-27 16:48:15 +00:00
  • 4b1ec1e1cc Added troubleshooting section includes a workaround fix for SSL CRL error Fixes StrawberryMaster/wayback-machine-downloader#33 Felipe 2025-10-08 11:33:50 +00:00
  • d7a63361e3 Use a FixedThreadPool for concurrent API calls Felipe 2025-09-24 21:05:22 +00:00
  • b1974a8dfa Refactor ConnectionPool to use SizedQueue for connection management and improve cleanup logic Felipe 2025-09-24 20:50:10 +00:00
  • 012b295aed Corrected wrong flag in example (#32) Huw Fulcher 2025-09-10 12:06:57 +01:00
  • dec9083b43 Fix: Fixed trivial mistake with function call adampweb 2025-08-13 21:59:38 +02:00
  • c517bd20d3 Actual retry implementation seems I pushed an older revision of this apparently Felipe 2025-09-04 19:16:52 +00:00
  • fc8d8a9441 Added retry command fixes [Feature request} Retry flag Fixes StrawberryMaster/wayback-machine-downloader#31 Felipe 2025-08-20 01:21:29 +00:00
  • fa306ac92b Bumped version v2.4.3 Felipe 2025-08-19 16:17:53 +00:00
  • 8c27aaebc9 Fix issue with index.html pages not loading we were rejecting empty paths, causing these files to be skipped. How did I miss this? Felipe 2025-08-19 16:16:24 +00:00
  • 40e9c9bb51 Bumped version v2.4.2 Felipe 2025-08-16 19:38:01 +00:00
  • 6bc08947b7 More aggressive sanitization this should deal with some of the issues we've seen, luckily. What a ride! Felipe 2025-08-12 18:55:00 -03:00
  • c731e0c7bd Bumped version v2.4.1 Felipe 2025-08-12 11:46:03 +00:00
  • 9fd2a7f8d1 Minor refactoring of HTML tag sanitization Felipe 2025-08-12 08:42:27 -03:00
  • 6ad312f31f Sanitizing HTML tags some sites contain tags *in* their URL, and fail to save on some devices like Windows Felipe 2025-08-05 23:44:34 +00:00
  • 62ea35daa6 Bumping version v2.4.0 Felipe 2025-08-04 21:23:48 +00:00
  • 1f4202908f Fixes for tidy_bytes admittedly not the cleanest way to do this, although it works for #25. Felipe 2025-07-31 12:58:22 -03:00
  • bed3f6101c Added missing gemspec file Felipe 2025-07-31 12:57:03 -03:00
  • 754df6b8d6 Merge pull request #27 from adampweb/master Felipe 2025-07-29 18:09:51 -03:00
  • 801fb77f79 Perf: Refactored a huge function into smaller subprocesses adampweb 2025-07-29 21:12:20 +02:00
  • e9849e6c9c Cleanup: I removed the obsolete options. The classic way provides more flexibility adampweb 2025-07-29 20:55:10 +02:00
  • bc868e6b39 Refactor tidy_bytes.rb Felipe 2025-07-29 10:10:56 -03:00
  • 2bf04aff48 Sanitize base_url and directory parameters this might be the cause of #25, at least from what it appears Felipe 2025-07-27 17:18:57 +00:00
  • 51becde916 Minor fix Felipe 2025-07-26 21:01:40 +00:00
  • c30ee73977 Sanitize file_id we were not consistently handling non-UTF-8 characters here, especially after commit e4487baafc. This also fixes #25 Felipe 2025-07-26 20:58:50 +00:00
  • d3466b3387 Bumping version normally I would've yanked the old gem, but that's not working here v2.3.12 Felipe 2025-07-22 12:41:26 +00:00
  • 0250579f0e Added missing file Felipe 2025-07-22 12:38:12 +00:00
  • 0663c1c122 Merge pull request #23 from adampweb/master Felipe 2025-07-21 14:44:43 -03:00
  • 93115f70ec Merge pull request #5 from adampweb/snyk-fix-88576ceadf7e0c41b63a2af504a3c8ae adampweb 2025-07-21 18:46:03 +02:00
  • 3d37ae10fd fix: Dockerfile to reduce vulnerabilities snyk-bot 2025-07-21 16:45:10 +00:00
  • bff10e7260 Initial implementation of a composite snapshot see issue #22. TBF Felipe 2025-07-21 15:30:49 +00:00
  • 3d181ce84c Bumped version v2.3.11 Felipe 2025-07-21 13:48:34 +00:00
  • 999aa211ae fix match filters Alfonso Corrado 2025-07-19 14:49:04 +02:00
  • ffdce7e4ec Exclude dev enviroment config adampweb 2025-07-20 17:14:09 +02:00
  • e4487baafc Fix: Handle default case in tidy_bytes adampweb 2025-07-20 17:13:36 +02:00
  • 82ff2de3dc Added brief note for users with both WMD gems here Felipe 2025-07-14 08:12:38 -03:00
  • fd329afdd2 Merge pull request #20 from underarchiver/rfc3968-url-validity-check Felipe 2025-07-11 10:55:12 -03:00
  • 038785557d Ability to recursively download across subdomains this is quite experimental. Fixes #15 but still needs more testing Felipe 2025-07-09 12:53:58 +00:00
  • 2eead8cc27 Bumping version v2.3.10 Felipe 2025-06-27 19:50:39 +00:00
  • 7e5cdd54fb Fix: path sanitizer and timestamp sorting errors cybercode3 2025-06-22 19:45:05 -05:00
  • 4160ff5e4a Bumping version v2.3.9 Felipe 2025-06-18 18:05:31 +00:00
  • f03d92a3c4 Prevent fetching off non RFC3968-compliant URLs underarchiver 2025-06-17 13:27:10 +02:00
  • 2490109cfe Merge pull request #17 from elidickinson/fix-exact-url Felipe 2025-06-15 22:18:40 -03:00
  • c3c5b8446a don’t append /* when —exact-url Eli Dickinson 2025-06-15 13:11:02 -04:00
  • 18357a77ed Correct file path and sanitization in Windows Not only we weren't normalizing the file directories, we were also agressively sanitizing incorrect characters, leading to some funny stuff on Windows. Fixes #16 Felipe 2025-06-15 13:48:11 +00:00
  • 3fdfd70fc1 Bump version v2.3.8 Felipe 2025-06-05 22:34:40 +00:00
  • 2bf74b4173 Merge pull request #14 from elidickinson/fix-bracket-urls Felipe 2025-06-03 23:12:07 -03:00
  • 79cbb639e7 Fix bug with archive urls containing square brackets Eli Dickinson 2025-06-03 16:36:03 -04:00
  • 071d208b31 Merge pull request #13 from elidickinson/master Felipe 2025-05-30 14:34:32 -03:00
  • 1681a12579 workaround for API only showing html files for some domains Eli Dickinson 2025-05-30 12:50:48 -04:00
  • f38756dd76 Correction for downloaded data folder if you downloaded content from example.org/*, it would be listed in a folder titled * instead of the sitename. See #6 (and thanks to elidickinson for pointing it out!) Felipe 2025-05-30 14:00:32 +00:00
  • 9452411e32 Added nil checks Felipe 2025-05-30 13:52:25 +00:00
  • 61e22cfe25 Bump versions v2.3.7 Felipe 2025-05-27 18:10:09 +00:00
  • 183ed61104 Attempt at fixing --all I honestly don't recall if this was implemented in the original code, and I'm guessing this worked at *some point* during this fork. It seems to work correctly now, however. See #6 and #11 Felipe 2025-05-27 17:17:34 +00:00
  • e6ecf32a43 Dockerfile test 2 Felipe 2025-05-21 21:34:36 -03:00
  • 375c6314ad Dockerfile test Felipe 2025-05-21 21:26:37 -03:00
  • 6e2739f5a8 Testing Felipe 2025-05-18 18:00:10 +00:00
  • caba6a665f Rough attempt to make this more efficient Felipe 2025-05-18 17:52:28 +00:00
  • ab4324c0eb Bumping to 2.3.6 v2.3.6 Felipe 2025-05-18 16:49:44 +00:00
  • e28d7d578b Experimental ability to rewrite URLs to local browsing Felipe 2025-05-18 16:44:52 +00:00
  • a7a25574cf Merge pull request #10 from adampweb/master Felipe 2025-05-15 08:50:33 -03:00
  • 23cc3d69b1 Merge pull request #9 from adampweb/feature/increase-performance Felipe 2025-05-15 08:50:04 -03:00
  • 01fa1f8c9f Merge pull request #2 from vitaly-zdanevich/patch-1 adampweb 2025-05-14 21:19:11 +02:00
  • d2f98d9428 Merge remote-tracking branch 'upstream/master' into feature/increase-performance adampweb 2025-05-14 15:41:07 +02:00
  • c7a5381eaf Using nproc in Bundler processes adampweb 2025-05-14 15:03:22 +02:00
  • 9709834e20 Merge pull request #8 from adampweb/master Felipe 2025-05-12 10:36:10 -03:00
  • 77998372cb Docker: If you load any component of the app before (or during) the Docker build process, it may cause failures adampweb 2025-05-11 20:05:00 +02:00
  • 2c789b7df6 Restructure Docker Compose config adampweb 2025-05-11 11:27:08 +02:00
  • 1ef8c14c48 Removed unused variable from if condition adampweb 2025-05-11 10:57:36 +02:00
  • 780e45343f Merge pull request #7 from adampweb/master Felipe 2025-05-10 11:34:07 -03:00
  • 42e6d62284 Merge remote-tracking branch 'upstream/master' adampweb 2025-05-09 20:17:01 +02:00
  • 543161d7fb Supplement of docs adampweb 2025-05-09 19:54:15 +02:00
  • 99a6de981e Env. vars: set default values and related docs adampweb 2025-05-09 19:38:39 +02:00
  • d85c880d23 Vulnerablity fix: adampweb 2025-05-09 18:32:47 +02:00
  • 917f4f8798 Bumping version v2.3.5 Felipe 2025-04-30 13:05:30 +00:00
  • 787bc2e535 Added missing configs Felipe 2025-04-30 13:05:21 +00:00
  • 4db13a7792 Fix --all-timestamps we were accidentally removing the timestamp prefix from file_id, rendering that option useless in 2.3.4. This should again now. This will fix #4 Felipe 2025-04-30 13:01:29 +00:00
  • 31d51728af Bump version v2.3.4 Felipe 2025-04-19 14:07:05 +00:00
  • febffe5de4 Added support for resuming incomplete downloads Felipe 2025-04-19 13:40:14 +00:00
  • 27dd619aa4 gzip support Felipe 2025-04-19 13:07:07 +00:00
  • 576298dca8 License fix Felipe 2025-04-19 13:05:09 +00:00
  • dc71d1d167 Merge pull request #3 from adampweb/master Felipe 2025-04-14 12:06:37 -03:00
  • 13e88ce04a README.md: add -v .:/websites Vitaly Zdanevich 2025-04-14 10:56:01 +04:00
  • c7fc7c7b58 README.md: add docker example without cloning the repo Vitaly Zdanevich 2025-04-14 10:43:49 +04:00
  • 5aebf83fca Add interactivity by CLI adampweb 2025-04-06 17:02:39 +02:00
  • b1080f0219 Keep secrets :) adampweb 2025-04-06 16:56:59 +02:00
  • dde36ea840 Merge branch 'StrawberryMaster:master' into master adampweb 2025-04-06 12:46:04 +02:00
  • acec026ce1 Using Docker Compose adampweb 2025-04-06 12:36:31 +02:00
  • ec3fd2dcaa Merge pull request #2 from adampweb/master Felipe 2025-04-02 22:52:25 -03:00
  • 6518ecf215 Install concurrent-ruby gem to avoid errors like cannot load such file -- concurrent-ruby adampweb 2025-04-02 15:23:24 +02:00
  • f5572d6129 Merge branch 'master' of https://github.com/adampweb/wayback-machine-downloader adampweb 2025-04-02 14:50:14 +02:00