Bump to 2.4.4

see `Results saved in /build/websites` but nothing is saved :(
Fixes StrawberryMaster/wayback-machine-downloader#34

README.md

@@ -9,7 +9,7 @@ Included here is partial content from other forks, namely those @ [ShiftaDeband]
 
 Download a website's latest snapshot:
 ```bash
-ruby wayback_machine_downloader https://example.com
+wayback_machine_downloader https://example.com
 ```
 Your files will save to `./websites/example.com/` with their original structure preserved.
 
@@ -27,6 +27,7 @@ To run most commands, just like in the original WMD, you can use:
 ```bash
 wayback_machine_downloader https://example.com
 ```
+Do note that you can also manually download this repository and run commands here by appending `ruby` before a command, e.g. `ruby wayback_machine_downloader https://example.com`.
 **Note**: this gem may conflict with hartator's wayback_machine_downloader gem, and so you may have to uninstall it for this WMD fork to work. A good way to know is if a command fails; it will list the gem version as 2.3.1 or earlier, while this WMD fork uses 2.3.2 or above.
 
 ### Step-by-step setup
@@ -63,15 +64,14 @@ docker build -t wayback_machine_downloader .
 docker run -it --rm wayback_machine_downloader [options] URL
 ```
 
-or the example without cloning the repo - fetching smallrockets.com until the year 2013:
+As an example of how this works without cloning this repo, this command fetches smallrockets.com until the year 2013:
 
 ```bash
 docker run -v .:/websites ghcr.io/strawberrymaster/wayback-machine-downloader:master wayback_machine_downloader --to 20130101 smallrockets.com
 ```
 
 ### 🐳 Using Docker Compose
-
+You can also use Docker Compose, which provides a lot of benefits for extending more functionalities (such as implementing storing previous downloads in a database):
-We can also use it with Docker Compose, which provides a lot of benefits for extending more functionalities (such as implementing storing previous downloads in a database):
 ```yaml
 # docker-compose.yml
 services:
@@ -120,6 +120,7 @@ STATE_DB_FILENAME = '.downloaded.txt'  # Tracks completed downloads
 | `-t TS`, `--to TS`  | Stop at timestamp |
 | `-e`, `--exact-url`     | Download exact URL only |
 | `-r`, `--rewritten`     | Download rewritten Wayback Archive files only |
+| `-rt`, `--retry NUM` | Number of tries in case a download fails (default: 1) |
 
 **Example** - Download files to `downloaded-backup` folder
 ```bash
@@ -165,6 +166,8 @@ ruby wayback_machine_downloader https://example.com --rewritten
 ```
 Useful if you want to download the rewritten files from the Wayback Machine instead of the original ones.
 
+---
+
 ### Filtering Content
 | Option | Description |
 |--------|-------------|
@@ -199,6 +202,8 @@ Or if you want to download everything except images:
 ruby wayback_machine_downloader https://example.com --exclude "/\.(gif|jpg|jpeg)$/i"
 ```
 
+---
+
 ### Performance
 | Option | Description |
 |--------|-------------|
@@ -213,10 +218,12 @@ Will specify the number of multiple files you want to download at the same time.
 
 **Example 2** - 300 snapshot pages:
 ```bash
-ruby wayback_machine_downloader https://example.com --snapshot-pages 300    
+ruby wayback_machine_downloader https://example.com --maximum-snapshot 300    
 ```
 Will specify the maximum number of snapshot pages to consider. Count an average of 150,000 snapshots per page. 100 is the default maximum number of snapshot pages and should be sufficient for most websites. Use a bigger number if you want to download a very large website.
 
+---
+
 ### Diagnostics
 | Option | Description |
 |--------|-------------|
@@ -235,6 +242,8 @@ ruby wayback_machine_downloader https://example.com --list
 ```
 It will just display the files to be downloaded with their snapshot timestamps and urls. The output format is JSON. It won't download anything. It's useful for debugging or to connect to another application.
 
+---
+
 ### Job management
 The downloader automatically saves its progress (`.cdx.json` for snapshot list, `.downloaded.txt` for completed files) in the output directory. If you run the same command again pointing to the same output directory, it will resume where it left off, skipping already downloaded files.
 
@@ -258,6 +267,47 @@ ruby wayback_machine_downloader https://example.com --keep
 ```
 This can be useful for debugging or if you plan to extend the download later with different parameters (e.g., adding `--to` timestamp) while leveraging the existing snapshot list.
 
+---
+
+## Troubleshooting
+
+### SSL certificate errors
+If you encounter an SSL error like:
+```
+SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get certificate CRL)
+```
+
+This is a known issue with **OpenSSL 3.6.0** when used with certain Ruby installations, and not a bug with this WMD work specifically. (See [ruby/openssl#949](https://github.com/ruby/openssl/issues/949) for details.)
+
+The workaround is to create a file named `fix_ssl_store.rb` with the following content:
+```ruby
+require "openssl"
+store = OpenSSL::X509::Store.new.tap(&:set_default_paths)
+OpenSSL::SSL::SSLContext::DEFAULT_PARAMS[:cert_store] = store
+```
+
+and run wayback-machine-downloader with:
+```bash
+RUBYOPT="-r./fix_ssl_store.rb" wayback_machine_downloader "http://example.com"
+```
+
+#### Verifying the issue
+You can test if your Ruby environment has this issue by running:
+
+```ruby
+require "net/http"
+require "uri"
+
+uri = URI("https://web.archive.org/")
+Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
+  resp = http.get("/")
+  puts "GET / => #{resp.code}"
+end
+```
+If this fails with the same SSL error, the workaround above will fix it.
+
+---
+
 ## 🤝 Contributing
 1. Fork the repository
 2. Create a feature branch

bin/wayback_machine_downloader

@@ -74,6 +74,10 @@ option_parser = OptionParser.new do |opts|
     options[:keep] = true
   end
 
+  opts.on("--rt", "--retry N", Integer, "Maximum number of retries for failed downloads (default: 3)") do |t|
+    options[:max_retries] = t
+  end
+
   opts.on("--recursive-subdomains", "Recursively download content from subdomains") do |t|
     options[:recursive_subdomains] = true
   end

lib/wayback_machine_downloader.rb

@@ -25,58 +25,90 @@ class ConnectionPool
   MAX_RETRIES = 3
 
   def initialize(size)
-    @size = size
+    @pool = SizedQueue.new(size)
-    @pool = Concurrent::Map.new
+    size.times { @pool << build_connection_entry }
-    @creation_times = Concurrent::Map.new
     @cleanup_thread = schedule_cleanup
   end
 
-  def with_connection(&block)
+  def with_connection
-    conn = acquire_connection
+    entry = acquire_connection
     begin
-      yield conn
+      yield entry[:http]
     ensure
-      release_connection(conn)
+      release_connection(entry)
     end
   end
 
   def shutdown
     @cleanup_thread&.exit
-    @pool.each_value { |conn| conn.finish if conn&.started? }
+    drain_pool { |entry| safe_finish(entry[:http]) }
-    @pool.clear
-    @creation_times.clear
   end
 
   private
 
   def acquire_connection
-    thread_id = Thread.current.object_id
+    entry = @pool.pop
-    conn = @pool[thread_id]
+    if stale?(entry)
-
+      safe_finish(entry[:http])
-    if should_create_new?(conn)
+      entry = build_connection_entry
-      conn&.finish if conn&.started?
-      conn = create_connection
-      @pool[thread_id] = conn
-      @creation_times[thread_id] = Time.now
     end
-
+    entry
-    conn
   end
 
-  def release_connection(conn)
+  def release_connection(entry)
-    return unless conn
+    if stale?(entry)
-    if conn.started? && Time.now - @creation_times[Thread.current.object_id] > MAX_AGE
+      safe_finish(entry[:http])
-      conn.finish
+      entry = build_connection_entry
-      @pool.delete(Thread.current.object_id)
+    end
-      @creation_times.delete(Thread.current.object_id)
+    @pool << entry
+  end
+
+  def stale?(entry)
+    http = entry[:http]
+    !http.started? || (Time.now - entry[:created_at] > MAX_AGE)
+  end
+
+  def build_connection_entry
+    { http: create_connection, created_at: Time.now }
+  end
+
+  def safe_finish(http)
+    http.finish if http&.started?
+  rescue StandardError
+    nil
+  end
+
+  def drain_pool
+    loop do
+      entry = begin
+        @pool.pop(true)
+      rescue ThreadError
+        break
+      end
+      yield(entry)
     end
   end
 
-  def should_create_new?(conn)
+  def cleanup_old_connections
-    return true if conn.nil?
+    entry = begin
-    return true unless conn.started?
+      @pool.pop(true)
-    return true if Time.now - @creation_times[Thread.current.object_id] > MAX_AGE
+    rescue ThreadError
-    false
+      return
+    end
+    if stale?(entry)
+      safe_finish(entry[:http])
+      entry = build_connection_entry
+    end
+    @pool << entry
+  end
+
+  def schedule_cleanup
+    Thread.new do
+      loop do
+        cleanup_old_connections
+        sleep CLEANUP_INTERVAL
+      end
+    end
   end
 
   def create_connection
@@ -89,27 +121,6 @@ class ConnectionPool
     http.start
     http
   end
-
-  def schedule_cleanup
-    Thread.new do
-      loop do
-        cleanup_old_connections
-        sleep CLEANUP_INTERVAL
-      end
-    end
-  end
-
-  def cleanup_old_connections
-    current_time = Time.now
-    @creation_times.each do |thread_id, creation_time|
-      if current_time - creation_time > MAX_AGE
-        conn = @pool[thread_id]
-        conn&.finish if conn&.started?
-        @pool.delete(thread_id)
-        @creation_times.delete(thread_id)
-      end
-    end
-  end
 end
 
 class WaybackMachineDownloader
@@ -117,7 +128,7 @@ class WaybackMachineDownloader
   include ArchiveAPI
   include SubdomainProcessor
 
-  VERSION = "2.4.3"
+  VERSION = "2.4.4"
   DEFAULT_TIMEOUT = 30
   MAX_RETRIES = 3
   RETRY_DELAY = 2
@@ -163,6 +174,7 @@ class WaybackMachineDownloader
     @recursive_subdomains = params[:recursive_subdomains] || false
     @subdomain_depth = params[:subdomain_depth] || 1
     @snapshot_at = params[:snapshot_at] ? params[:snapshot_at].to_i : nil
+    @max_retries = params[:max_retries] ? params[:max_retries].to_i : MAX_RETRIES
 
     # URL for rejecting invalid/unencoded wayback urls
     @url_regexp = /^(([A-Za-z][A-Za-z0-9+.-]*):((\/\/(((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=]))+)(:([0-9]*))?)(((\/((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)*))*)))|((\/(((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)+)(\/((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)*))*)?))|((((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)+)(\/((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)*))*)))(\?((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)|\/|\?)*)?(\#((([A-Za-z0-9._~-])|(%[ABCDEFabcdef0-9][ABCDEFabcdef0-9])|([!$&'('')'*+,;=])|:|@)|\/|\?)*)?)$/
@@ -193,7 +205,8 @@ class WaybackMachineDownloader
       @directory
     else
       # ensure the default path is absolute and normalized
-      File.expand_path(File.join('websites', backup_name))
+      cwd = Dir.pwd
+      File.expand_path(File.join(cwd, 'websites', backup_name))
     end
   end
 
@@ -277,53 +290,58 @@ class WaybackMachineDownloader
       page_index = 0
       batch_size = [@threads_count, 5].min
       continue_fetching = true
+      fetch_pool = Concurrent::FixedThreadPool.new([@threads_count, 1].max)
+      begin
+        while continue_fetching && page_index < @maximum_pages
+          # Determine the range of pages to fetch in this batch
+          end_index = [page_index + batch_size, @maximum_pages].min
+          current_batch = (page_index...end_index).to_a
 
-      while continue_fetching && page_index < @maximum_pages
+          # Create futures for concurrent API calls
-        # Determine the range of pages to fetch in this batch
+          futures = current_batch.map do |page|
-        end_index = [page_index + batch_size, @maximum_pages].min
+            Concurrent::Future.execute(executor: fetch_pool) do
-        current_batch = (page_index...end_index).to_a
+              result = nil
-
+              @connection_pool.with_connection do |connection|
-        # Create futures for concurrent API calls
+                result = get_raw_list_from_api("#{@base_url}/*", page, connection)
-        futures = current_batch.map do |page|
+              end
-          Concurrent::Future.execute do
+              result ||= []
-            result = nil
+              [page, result]
-            @connection_pool.with_connection do |connection|
-              result = get_raw_list_from_api("#{@base_url}/*", page, connection)
-            end
-            result ||= []
-            [page, result]
-          end
-        end
-
-        results = []
-
-        futures.each do |future|
-          begin
-            results << future.value
-          rescue => e
-            puts "\nError fetching page #{future}: #{e.message}"
-          end
-        end
-
-        # Sort results by page number to maintain order
-        results.sort_by! { |page, _| page }
-
-        # Process results and check for empty pages
-        results.each do |page, result|
-          if result.nil? || result.empty?
-            continue_fetching = false
-            break
-          else
-            mutex.synchronize do
-              snapshot_list_to_consider.concat(result)
-              print "."
             end
           end
+
+          results = []
+
+          futures.each do |future|
+            begin
+              results << future.value
+            rescue => e
+              puts "\nError fetching page #{future}: #{e.message}"
+            end
+          end
+
+          # Sort results by page number to maintain order
+          results.sort_by! { |page, _| page }
+
+          # Process results and check for empty pages
+          results.each do |page, result|
+            if result.nil? || result.empty?
+              continue_fetching = false
+              break
+            else
+              mutex.synchronize do
+                snapshot_list_to_consider.concat(result)
+                print "."
+              end
+            end
+          end
+
+          page_index = end_index
+
+          sleep(RATE_LIMIT) if continue_fetching
         end
-
+      ensure
-        page_index = end_index
+        fetch_pool.shutdown
-
+        fetch_pool.wait_for_termination
-        sleep(RATE_LIMIT) if continue_fetching
       end
     end
 
@@ -638,13 +656,13 @@ class WaybackMachineDownloader
       end
 
       # URLs in HTML attributes
-      rewrite_html_attr_urls(content)
+      content = rewrite_html_attr_urls(content)
       
       # URLs in CSS
-      rewrite_css_urls(content)
+      content = rewrite_css_urls(content)
       
       # URLs in JavaScript
-      rewrite_js_urls(content)
+      content = rewrite_js_urls(content)
       
       # for URLs in HTML attributes that start with a single slash
       content.gsub!(/(\s(?:href|src|action|data-src|data-url)=["'])\/([^"'\/][^"']*)(["'])/i) do
@@ -934,9 +952,9 @@ class WaybackMachineDownloader
       end
 
     rescue StandardError => e
-      if retries < MAX_RETRIES
+      if retries < @max_retries
         retries += 1
-        @logger.warn("Retry #{retries}/#{MAX_RETRIES} for #{file_url}: #{e.message}")
+        @logger.warn("Retry #{retries}/#{@max_retries} for #{file_url}: #{e.message}")
         sleep(RETRY_DELAY * retries)
         retry
       else

wayback_machine_downloader.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name        = "wayback_machine_downloader_straw"
-  s.version     = "2.4.3"
+  s.version     = "2.4.4"
   s.executables << "wayback_machine_downloader"
   s.summary     = "Download an entire website from the Wayback Machine."
   s.description = "Download complete websites from the Internet Archive's Wayback Machine. While the Wayback Machine (archive.org) excellently preserves web history, it lacks a built-in export functionality; this gem does just that, allowing you to download entire archived websites. (This is a significant rewrite of the original wayback_machine_downloader gem by hartator, with enhanced features and performance improvements.)"