Compare commits

..
49 Commits
Author SHA1 Message Date
death-claw 745bc17606 v5 wip 2023-10-13 00:00:01 +01:00
death-claw 04e37c83d7 v5 wip 2023-10-12 01:15:13 +01:00
death-claw 9bb865d98e v5 wip 2023-10-11 21:16:57 +01:00
death-claw 1625c70ce0 v5 wip 2023-10-11 00:50:03 +01:00
death-claw 63bc3ed3cc v5 wip 2023-10-10 19:17:16 +01:00
death-claw b1565e3a71 v5 wip 2023-10-09 08:46:06 +01:00
death-claw b1ef2243a0 v5 wip 2023-10-08 23:54:47 +01:00
death-claw 1d1c8918f0 v5 wip 2023-10-08 20:32:09 +01:00
death-claw cc0972adbc v5 wip 2023-10-08 09:30:06 +01:00
death-claw d9e8e08adb v5 wip 2023-10-06 09:31:02 +01:00
death-claw c08c430160 v5 wip 2023-10-04 00:39:09 +01:00
death-claw 4572ca61a3 v5 wip 2023-10-03 00:58:20 +01:00
death-claw ca23c80c24 v5 wip 2023-10-02 00:33:06 +01:00
l1xanderandGitHub f0c2de829a Fix for images on Pimpandhost throwing exception (#127)
Attempts to fix the following exception:


Failed to download http://pimpandhost.com/image/39446205-medium.html
me.mnlr.vripper.exception. DownloadException: me.mnlr.vripper.exception.HostException: Xpath //img[contains(@class, 'original')]' cannot be found in 'http://pimpandhost.com/image/39446205-medium.html' at me.mnlr.vripper.download.ImageDownloadRunnable.download(Image Download Runnable.kt:88)
at me.mnlr.vripper.download.ImageDownloadRunnable.run(ImageDownloadRunnable.kt:151)
at net.jodah.failsafe.Functions.lambda$toSupplier$10(Functions.java:262)
at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48)
at net.jodah.failsafe.Retry Policy Executor.lambda$supply$0(RetryPolicyExecutor.java:66)
at net.jodah.failsafe.Execution.executeSync(Execution.java:128)
at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:379)
at net.jodah.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:212)
at me.mnlr.vripper.download.DownloadService.schedule$lambda$28 (DownloadService.kt:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: me.mnlr.vripper.exception.HostException: Xpath //img[contains(@class, 'original')]' cannot be found in 'http://pimpandhost.com/image/39446205-medium.html'
at me.mnlr.vripper.host.PimpandhostHost.resolve(PimpandhostHost.kt:46)
at me.mnlr.vripper.host.Host.download Internal(Host.kt:62)
at me.mnlr.vripper.download.ImageDownloadRunnable.download (ImageDownloadRunnable.kt:72)
11 more
2023-09-22 10:04:19 +01:00
death-claw 08fcd7a87d version bump 2023-09-14 22:17:52 +01:00
death-clawandGitHub f39c7a15c7 fixes #123 (#126) 2023-09-14 21:53:49 +01:00
death-clawandGitHub 1c239d435b fixes #122 (#125) 2023-09-14 21:53:35 +01:00
death-clawandGitHub d8cba97d05 fixes #120 (#124) 2023-09-14 21:52:08 +01:00
death-clawandGitHub a362a3fa7f fixes #118 (#119) 2023-09-03 14:04:50 +01:00
death-clawandGitHub 4163f2a53f fixes #116 (#117) 2023-09-03 14:02:44 +01:00
death-claw e556dd1804 fix packaging 2023-09-01 15:24:32 +01:00
death-claw 96ad55b930 fix packaging 2023-09-01 15:17:33 +01:00
death-claw 9dc1a10255 fix packaging 2023-09-01 15:02:03 +01:00
death-claw db4a041bc4 fix packaging 2023-09-01 14:48:17 +01:00
death-claw 661f591bf5 fix packaging 2023-09-01 14:35:44 +01:00
death-claw e81bcfbfee fix packaging 2023-09-01 14:27:27 +01:00
death-claw e8ba07aebd version bump 2023-09-01 13:57:02 +01:00
death-clawandGitHub 9930226dc0 Enhance packaging (#114) 2023-09-01 13:54:51 +01:00
death-clawandGitHub 48491767cb remove threads when auto queue activates (#112) 2023-09-01 13:05:16 +01:00
death-clawandGitHub 49229b9dac open gallery folder option missing on right click (#111) 2023-09-01 01:01:27 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ecf3287e61 Bump socket.io-parser from 4.2.2 to 4.2.4 in /vripper-web-ui (#108)
Bumps [socket.io-parser](https://github.com/socketio/socket.io-parser) from 4.2.2 to 4.2.4.
- [Release notes](https://github.com/socketio/socket.io-parser/releases)
- [Changelog](https://github.com/socketio/socket.io-parser/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io-parser/compare/4.2.2...4.2.4)

---
updated-dependencies:
- dependency-name: socket.io-parser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-01 00:10:35 +01:00
death-clawandGitHub 206b305ae6 have an option to save temp files on different drive (#110) 2023-09-01 00:09:49 +01:00
death-clawandGitHub 70e7a87189 dependencies update (#109) 2023-08-31 23:28:59 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5aef1c0a89 Bump word-wrap from 1.2.3 to 1.2.4 in /vripper-web-ui (#106)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 22:43:30 +01:00
death-claw 7745abbb8a Update package.yml and version bump 2023-05-26 00:22:12 +01:00
death-clawandGitHub e577dba7c9 #100 wip (#101) 2023-05-24 22:20:02 +01:00
death-clawandGitHub ed63d5c029 Feature/issue 97/enhance lock file mechanism (#99)
* fixes #97

* fixes #97
2023-05-24 20:42:25 +01:00
death-clawandGitHub 2aa82a73a0 fixes #97 (#98) 2023-05-24 20:27:31 +01:00
death-clawandGitHub cbe5800c08 Update and rename maven-publish.yml to release.yml 2023-05-21 17:28:38 +01:00
death-clawandGitHub b200226060 Update maven-publish.yml 2023-05-21 16:56:53 +01:00
death-clawandGitHub 0d22b054c9 v4.1.0 (#96) 2023-05-21 16:42:11 +01:00
death-clawandGitHub 9d18dc8897 Update maven-publish.yml 2023-05-21 16:38:06 +01:00
death-claw f42d412b86 fixes #89 (#95) 2023-05-21 15:55:42 +01:00
death-clawandGitHub f8370801c2 fixes #89 (#94) 2023-05-21 13:18:23 +01:00
death-clawandGitHub 51dcaa1946 fixes #88 (#93) 2023-05-14 21:44:44 +01:00
death-clawandGitHub a833a0b1b2 Update maven-publish.yml 2023-05-13 17:20:40 +01:00
death-clawandGitHub 79739b80f9 fixes #90 (#92) 2023-05-13 17:16:00 +01:00
death-claw 26cf60b741 version bump 2023-05-06 21:26:39 +01:00
death-claw bc45deed2a resolves #86 2023-05-06 21:22:18 +01:00
353 changed files with 22735 additions and 15905 deletions
-34
View File
@@ -1,34 +0,0 @@
# This workflow will build a package using Maven and then publish it to GitHub packages when a release is created
# For more information see: https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#apache-maven-with-a-settings-path
name: Maven Package
on:
release:
types: [created]
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Publish to GitHub Packages Apache Maven
run: mvn deploy -B -s $GITHUB_WORKSPACE/settings.xml
env:
GITHUB_TOKEN: ${{ github.token }}
+55
View File
@@ -0,0 +1,55 @@
name: Package
on:
workflow_dispatch:
jobs:
build:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build with Maven
run: mvn -B install --file pom.xml
- name: Rename GUI artifact
run: mv vripper-gui/target/vripper-gui-*.jar vripper-gui/target/vripper-gui.jar
- name: Rename WEB artifact
run: mv vripper-web/target/vripper-web-*.jar vripper-web/target/vripper-web.jar
- if: matrix.os == 'ubuntu-latest'
name: Package Linux binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage --type deb "@jpackage.cfg" "@jpackage-linux.cfg"
jpackage --type rpm "@jpackage.cfg" "@jpackage-linux.cfg"
ls -la dist
- if: matrix.os == 'windows-latest'
name: Package Windows binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage "@jpackage.cfg" "@jpackage-windows.cfg"
dir dist
- if: matrix.os == 'macos-latest'
name: Package macOS binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage "@jpackage.cfg" "@jpackage-macos.cfg"
ls -la dist
+144
View File
@@ -0,0 +1,144 @@
name: Release
on:
release:
types: [created]
jobs:
build:
strategy:
matrix:
os: [ ubuntu-latest, windows-latest, macos-latest ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build Core Jar
run: mvn -B -q install --file vripper-core/pom.xml
- name: Build GUI Jar
run: mvn -B -q install --file vripper-gui/pom.xml
- name: Rename GUI Jar
run: mv vripper-gui/target/vripper-gui-${{ github.event.release.tag_name }}.jar vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar
- if: matrix.os == 'ubuntu-latest'
name: Upload GUI Jar
uses: softprops/action-gh-release@v1
with:
files: vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar
# Start building WEB jar in ubuntu only
- if: matrix.os == 'ubuntu-latest'
name: Build WEB Jar
run: |
mvn -B -q install --file vripper-web-ui/pom.xml
mvn -B -q install --file vripper-web/pom.xml
- if: matrix.os == 'ubuntu-latest'
name: Rename WEB Jar
run: mv vripper-web/target/vripper-web-${{ github.event.release.tag_name }}.jar vripper-web/target/vripper-noarch-web-${{ github.event.release.tag_name }}.jar
- if: matrix.os == 'ubuntu-latest'
name: Upload WEB Jar
uses: softprops/action-gh-release@v1
with:
files: vripper-web/target/vripper-noarch-web-${{ github.event.release.tag_name }}.jar
# End building WEB jar in ubuntu only
- name: Prepare Packaging
run: |
cp vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar jpackage/jar/vripper-gui.jar
- if: matrix.os == 'ubuntu-latest'
name: Package Linux
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.png
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-linux.cfg" --type deb
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-linux.cfg" --type rpm
mv dist/vripper-${{ github.event.release.tag_name }}-1.x86_64.rpm dist/vripper-linux-${{ github.event.release.tag_name }}.x86_64.rpm
mv dist/vripper_${{ github.event.release.tag_name }}-1_amd64.deb dist/vripper-linux-${{ github.event.release.tag_name }}_amd64.deb
- if: matrix.os == 'windows-latest'
name: Package Windows
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.ico
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-windows.cfg" --type msi
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-windows.cfg" --type exe
cd dist
ren VRipper-${{ github.event.release.tag_name }}.msi vripper-windows-installer-${{ github.event.release.tag_name }}.msi
ren VRipper-${{ github.event.release.tag_name }}.exe vripper-windows-installer-${{ github.event.release.tag_name }}.exe
- if: matrix.os == 'macos-latest'
name: Package macOS
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.icns
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-macos.cfg" --type pkg
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-macos.cfg" --type dmg
mv dist/VRipper-${{ github.event.release.tag_name }}.pkg dist/vripper-macos-${{ github.event.release.tag_name }}.pkg
mv dist/VRipper-${{ github.event.release.tag_name }}.dmg dist/vripper-macos-${{ github.event.release.tag_name }}.dmg
- if: matrix.os == 'ubuntu-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper'
filename: 'vripper-linux-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'windows-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper'
filename: 'vripper-windows-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'macos-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper.app'
filename: 'vripper-macos-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'ubuntu-latest'
name: Upload DEB and RPM package for Linux
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-linux-${{ github.event.release.tag_name }}.x86_64.rpm
jpackage/dist/vripper-linux-${{ github.event.release.tag_name }}_amd64.deb
jpackage/dist/vripper-linux-portable-${{ github.event.release.tag_name }}.zip
- if: matrix.os == 'windows-latest'
name: Upload packages for Windows
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-windows-installer-${{ github.event.release.tag_name }}.msi
jpackage/dist/vripper-windows-installer-${{ github.event.release.tag_name }}.exe
jpackage/dist/vripper-windows-portable-${{ github.event.release.tag_name }}.zip
- if: matrix.os == 'macos-latest'
name: Upload package for macOS
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-macos-${{ github.event.release.tag_name }}.pkg
jpackage/dist/vripper-macos-${{ github.event.release.tag_name }}.dmg
jpackage/dist/vripper-macos-portable-${{ github.event.release.tag_name }}.zip
-763
View File
@@ -1,763 +0,0 @@
# Changelog
## [4.0.0] - 2023-05-03
- Drop electron app
- Use javafx for the desktop app
- Re-written in kotlin
- Several enhancements
### Changed
## [3.5.4] - 2021-05-29
### Changed
- Fix download queue bug
## [3.5.3] - 2021-05-24
### Changed
- Update donation infos
## [3.5.1] - 2021-05-22
### Changed
- Bug fix
## [3.5.0] - 2021-05-21
### Changed
- Download queue bug fix
- Save window size and position
- Resize confirmation box
- Enhance metadata interruption
- Add Added on and Order columns
## [3.4.2] - 2021-05-10
### Changed
- Fix potential issue with imagebam host
- Enhance download queue logic
- Code source enhancements
- Fix app icons
- Fix race condition bug
## [3.3.8] - 2021-05-05
### Changed
- Fix imagebam host
- Fix font path
## [3.3.6] - 2021-03-20
### Changed
- Optimize build size
## [3.3.5] - 2021-03-18
### Changed
- Bug fixe
## [3.3.4] - 2021-03-18
### Changed
- Bug fixes
## [3.3.3] - 2021-03-13
### Changed
- Remove unnecessary event logs
## [3.3.2] - 2021-03-11
### Changed
- Fix a bug with event logs
- Change event names and add some icons
## [3.3.1] - 2021-02-18
### Added
- Add about menu
### Changed
- Fix README.md
## [3.3.0] - 2021-02-16
### Changed
- Add support for proxies
- Add a Event Logs
- Fix bugs with the database
## [3.2.3] - 2021-01-13
### Changed
- Finished posts are not cleared from the UI
## [3.2.2] - 2021-01-13
### Changed
- Set total download to max value
## [3.2.1] - 2021-01-12
### Added
- Add settings for connection timeout
- Add settings for maximum attempts
### Changed
- Enhance posts rename logic
- Dependencies update
- Switch reactive from rx-java to reactor
## [3.2.0] - 2020-11-16
### Changed
- Fix build issue
## [3.1.0] - 2020-11-16
### Changed
- UI revamp
- Fix bugs
## [3.0.11] - 2020-10-31
### Added
- Support for vipr.im
## [3.0.10] - 2020-10-04
### Changed
- Fix bug on settings update
## [3.0.9] - 2020-09-27
### Changed
- Revert to appending post id to download folders
- Fix a bug that lead to adding duplicate posts
- Remove old unused settings
## [3.0.8] - 2020-08-22
### Changed
- Bug fix which breaks concurrent downloads
## [3.0.7] - 2020-08-22
### Changed
- Fix MacOs build
## [3.0.6] - 2020-08-18
### Changed
- Fix TurboImageHost
## [3.0.5] - 2020-08-17
### Changed
- Update packages
- Host class refactoring
## [3.0.4] - 2020-08-16
### Changed
- Improve rename UI
## [3.0.3] - 2020-08-16
### Changed
- Fix bugs with electron app
## [3.0.2] - 2020-08-15
### Changed
- Fix bugs with download queue
## [3.0.1] - 2020-08-03
### Changed
- Fix bugs with download queue
## [3.0.0] - 2020-08-03
### Changed
- Major rewrites
- Introducing a proper database for persistence
## [2.12.1] - 2020-05-22
### Changed
- Fix alternative name bug
## [2.12.0] - 2020-05-21
### Changed
- Use new user hash from the api
## [2.11.7] - 2020-05-08
### Changed
- Fix bug with paths creation
- Move Open Link button
- Rename rename all title
## [2.11.6] - 2020-05-04
### Changed
- PNG filenames are now correctly named
- Fix issue with special characters in post titles
## [2.11.5] - 2020-05-03
### Changed
- Reduce rows buffer size
## [2.11.4] - 2020-05-03
### Changed
- Code cleanup
- Dependencies update
## [2.11.3] - 2020-04-30
### Changed
- Source code restructure
- UI fixes and enhancements
### Added
- Add menu entry to rename a gallery to first alternative title
- Add menu on toolbar for massive rename to first alternative title
## [2.11.2] - 2020-04-26
### Changed
- Bug fixes
## [2.11.1] - 2020-04-25
### Changed
- Code source maintenance
### Added
- Added option to rename gallery
## [2.11.0] - 2020-03-29
### Changed
- Upgrade Electron to v8
- Upgrade Angular to v9
- App settings are now portable in a json file
- General code source maintenance
### Added
- Added Support for pixroute.com
- Added grab post title from thread option
## [2.10.8] - 2020-01-25
### Changed
- Fix app url
## [2.10.7] - 2020-01-25
### Changed
- Drop SockJs and switch to WebSocket
- Autoreconnect front to back
## [2.10.6] - 2020-01-19
### Changed
- Limit cache size
## [2.10.5] - 2020-01-18
### Changed
- Bug fixes
## [2.10.4] - 2020-01-18
### Changed
- Zero value for total max download will disable the limit
- Add system notification
- Bug fixes
## [2.10.3] - 2020-01-16
### Changed
- Some bug fixes
- Enhance download queue logic
- Clear cache button
- Fix the partial status
- Sanitize URLs with spaces
### Added
- Support for Postimg host
- Support for Imagevenue host
## [2.10.2] - 2019-12-29
### Changed
- Upgrade electron to v7
- Bug fix
## [2.10.1] - 2019-12-27
### Changed
- Bug fix
## [2.10.0] - 2019-12-27
### Changed
- Fix spring dependency bug
### Added
- Add support for imgspice
## [2.9.0] - 2019-12-27
### Changed
- Fix build issue
## [2.9.0] - 2019-12-27
### Changed
- Decrease timeout for fast fail
### Added
- Allow many hosts to run at the same time
- Add title to gallery
- Add global concurrent downloads
- Add counter badge in link collector
## [2.8.1] - 2019-12-24
### Changed
- Fix issue with path creation
## [2.8.0] - 2019-12-24
### Added
- Photo gallery feature
## [2.7.1] - 2019-12-21
### Added
- Add version to title bar
## [2.7.0] - 2019-12-21
### Changed
- Add missing icon
### Added
- Add support for pixxxels
## [2.6.0] - 2019-12-21
### Changed
- Disable download button when nothing selected in Link collector grab menu
- Better quality app icon
- Implement context menu on download tab
- Add stop remove and start button to context menu
- Link will not be removed automatically from link collector tab
- Some UI fixes
## [2.5.1] - 2019-12-09
### Changed
- Disable download button when nothing selected in Link collector grab menu
### Added
- Add hosts column in Link collector grab menu
## [2.5.0] - 2019-12-08
### Changed
- Change loading screen
- UI Fixes
- Fix bug with previews
### Added
- Add link collector feature
- Add support for multiple links at a time
## [2.4.1] - 2019-11-26
### Changed
- Increase attempts for establishing ws connection between front and back end
- Change default download folder to Home folder (installer), App folder (portable)
## [2.4.0] - 2019-11-23
### Changed
- Pimpandhost Host support
## [2.3.2] - 2019-11-23
### Changed
- Fix a bug with the scan component
## [2.3.1] - 2019-11-22
### Changed
- Fix connexion reset issue
- Fix Post Status when autostart option is selected
- Add stop all button
## [2.3.0] - 2019-11-21
### Changed
- UI enhancements and overhaul
- Stability fixes with HTTP connections
- MacOS official support
- Use local sockjs and material icons font
## [2.2.5] - 2019-11-09
### Changed
- Exclude thumbs download for imagezilla and pixhost
- Change retry on fail logic (may help avoid having download errors)
## [2.2.4] - 2019-10-05
### Added
- AppImage target for linux
## [2.2.3] - 2019-10-01
### Changed
- Strip libjvm.so for debian
## [2.2.2] - 2019-10-01
### Changed
- Update product name
## [2.2.1] - 2019-10-01
### Changed
- Update dependency list for deb package
## [2.2.0] - 2019-10-01
### Added
- Embedd JRE within the app
### Changed
- Fix 'Start' on pending items
- Upgrade to Java 11
## [2.1.0] - 2019-09-30
### Added
- Add a create a subfolder per thread option
## [2.0.4] - 2019-09-27
### Changed
- Fix some issues with ImxHost
## [2.0.3] - 2019-09-23
### Changed
- Update deb dependencies
## [2.0.2] - 2019-09-23
### Changed
- Add portable mode
## [2.0.1] - 2019-09-22
### Changed
- Change app name
## [2.0.0] - 2019-09-22
### Added
- Add sub folder option
- Add clear finished option
### Changed
- UI enhancements
- Bug fixes
- Change data and log paths
## [1.6.5] - 2019-09-18
### Changed
- electron update
## [1.6.4] - 2019-09-18
### Changed
- Change menu item name
## [1.6.3] - 2019-09-18
### Changed
- Better multithread management
- Fix partial state UI
## [1.6.2] - 2019-09-17
### Changed
- Rework UI colors
- Fix scan input bug
## [1.6.1] - 2019-09-16
### Changed
- Rework UI
- Fix clipboard monitoring
## [1.6.0] - 2019-09-15
### Changed
- Rework the UI
- Add a dark theme
- Fix the force ordering bug
- Other bug fixes
## [1.5.2] - 2019-09-01
### Added
- imx.to host fix
## [1.5.1] - 2019-08-24
### Added
- Bug fixes
## [1.5.0] - 2019-08-24
### Added
- Display logged in user
- Add post details when clicking on main list items (Status, Post URL, Host)
- Add open download location on main list items menu
- Add support for multi-post threads (Show a menu to select posts to download)
- Add link to image from details list
## [1.4.10] - 2019-08-15
### Changed
- Fix a rename issue on windows
## [1.4.9] - 2019-08-13
### Changed
- Update Built-By to be generic
## [1.4.8] - 2019-08-13
### Changed
- Add option to force image ordering
## [1.4.7] - 2019-08-13
### Changed
- Fixed a bug in image filename extensions
## [1.4.6] - 2019-08-06
### Changed
- Fixed a bug when creating image file names
- Remove unnecessary logs
- Fix data.json init
## [1.4.5] - 2019-08-03
### Changed
- Add min width and height for electron app
### Added
- Add download speed to status bar
## [1.4.4] - 2019-08-01
### Changed
- Fix names for ImgboxHost
- Enhance UI
- Minor fixes
### Added
- Add start and stop button
## [1.4.3] - 2019-07-14
### Changed
- Update some messages
- Fix scrolling issue
## [1.4.2] - 2019-07-14
### Changed
- Fix layout issue in web mode
## [1.4.1] - 2019-07-14
### Changed
- Change title's font
### Added
- Add borders
## [1.4.0] - 2019-07-14
### Changed
- Better error handling on startup
- Better support for linux AppImage
### Added
- Clear all completed
- Add clipboard support
- Add option to remove all posts
- Add confirmation on remove
## [1.3.0-rc0] - 2019-07-04
### Changed
- Change default setting for autostart false -> true
- Change default setting for max concurrent downloads 1 -> 4
### Added
- Desktop app built with electron
- Add the option to remove a post
## [1.2.0] - 2019-06-23
### Added
- Add a shutdown button to gracefully shutdown the app
- Open and redirect the browser to the app address on startup
- Add the option to run the app in headless mode (-Djava.awt.headless=true)
## [1.1.1] - 2019-06-22
### Changed
- Fix a stability issue with progress updates when the app is runninh for a long time
## [1.0.0] - 2019-06-22
### Added
- Add real time communication for progress updates via websocket (SockJS)
- Add settings for download path, max concurrent download, enable/disable integration with vipergirls
- Add integration with vipergitls.to (possibility to auto leave thanks after grab post)
- Add support for acidimg.cc, imagezilla.net, imgbox.com, imx.to, pixhost.to, turboimagehost.com
+2 -2
View File
@@ -12,8 +12,8 @@ To build run the following:
Build artifact is located under
vripper-project\vripper-gui\target\vripper-gui-4.0.0.jar
vripper-project\vripper-gui\target\vripper-gui-{{version}}.jar
Copy the artifact into any other folder and run:
java -jar vripper-gui-4.0.0.jar
java -jar vripper-gui-{{version}}.jar
Binary file not shown.
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 303 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

View File
+2
View File
@@ -0,0 +1,2 @@
--type app-image
--java-options "-Dbase.dir=${user.dir}"
+7
View File
@@ -0,0 +1,7 @@
--icon icon.png
--java-options "-Dbase.dir=${user.home}/.config"
--linux-package-name vripper
--linux-app-release 1
--linux-menu-group Utility
--linux-app-category java
--linux-shortcut
+4
View File
@@ -0,0 +1,4 @@
--icon icon.icns
--java-options "-Dbase.dir=${user.home}/.config"
--mac-package-identifier me.mnlr.vripper.vripper-gui
--mac-package-name VRipper
+8
View File
@@ -0,0 +1,8 @@
--icon icon.ico
--java-options "-Dbase.dir=${user.home}"
--win-dir-chooser
--win-menu
--win-per-user-install
--win-shortcut
--win-shortcut-prompt
--win-update-url https://github.com/death-claw/vripper-project/releases
+8
View File
@@ -0,0 +1,8 @@
--input jar
--main-jar vripper-gui.jar
--add-modules java.base,java.desktop,java.sql,jdk.unsupported,jdk.crypto.ec,java.security.jgss
--jlink-options "--strip-native-commands --strip-debug --no-man-pages --no-header-files --compress=2"
--name VRipper
--description "Image ripper tool for vipergirls"
--dest dist
--vendor "death-claw"
+128 -2
View File
@@ -7,11 +7,137 @@
<artifactId>vripper-project</artifactId>
<modules>
<module>vripper-core</module>
<!-- <module>vripper-web-ui</module>-->
<!-- <module>vripper-web</module>-->
<module>vripper-gui</module>
</modules>
<packaging>pom</packaging>
<version>4.0.0</version>
<version>4.4.0</version>
<properties>
<maven.deploy.skip>true</maven.deploy.skip>
<java.version>17</java.version>
<maven.deploy.skip>false</maven.deploy.skip>
<timestamp>${maven.build.timestamp}</timestamp>
<maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss</maven.build.timestamp.format>
<maven.compiler.target>17</maven.compiler.target>
<maven.compiler.source>17</maven.compiler.source>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
<version>1.4.11</version>
</dependency>
<dependency>
<artifactId>kotlin-stdlib</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>1.9.10</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core-jvm</artifactId>
<version>1.7.3</version>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-jdbc</artifactId>
<version>0.41.1</version>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-java-time</artifactId>
<version>0.41.1</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>3.5.10</version>
</dependency>
<dependency>
<groupId>io.insert-koin</groupId>
<artifactId>koin-core-jvm</artifactId>
<version>3.4.2</version>
</dependency>
<dependency>
<artifactId>jackson-module-kotlin</artifactId>
<groupId>com.fasterxml.jackson.module</groupId>
<version>2.15.2</version>
</dependency>
<dependency>
<artifactId>h2</artifactId>
<groupId>com.h2database</groupId>
<version>2.2.224</version>
</dependency>
<dependency>
<artifactId>httpclient</artifactId>
<groupId>org.apache.httpcomponents</groupId>
<version>4.5.14</version>
</dependency>
<dependency>
<artifactId>htmlcleaner</artifactId>
<groupId>net.sourceforge.htmlcleaner</groupId>
<version>2.26</version>
</dependency>
<dependency>
<artifactId>failsafe</artifactId>
<groupId>net.jodah</groupId>
<version>2.4.4</version>
</dependency>
<dependency>
<artifactId>caffeine</artifactId>
<groupId>com.github.ben-manes.caffeine</groupId>
<version>3.1.8</version>
</dependency>
<dependency>
<artifactId>jackson-databind</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>5.13.0</version>
</dependency>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>1.9.10</version>
</dependency>
<dependency>
<artifactId>jcl-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
<version>1.7.32</version>
</dependency>
<dependency>
<artifactId>httpclient5</artifactId>
<groupId>org.apache.httpcomponents.client5</groupId>
<version>5.2.1</version>
</dependency>
<dependency>
<artifactId>commons-codec</artifactId>
<groupId>commons-codec</groupId>
<version>1.16.0</version>
</dependency>
<dependency>
<artifactId>liquibase-core</artifactId>
<groupId>org.liquibase</groupId>
<version>4.24.0</version>
</dependency>
</dependencies>
</dependencyManagement>
</project>
+54 -58
View File
@@ -4,65 +4,67 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
<relativePath/>
<version>3.0.6</version>
<groupId>me.mnlr</groupId>
<artifactId>vripper-project</artifactId>
<version>4.4.0</version>
</parent>
<groupId>me.mnlr.vripper</groupId>
<artifactId>vripper-core</artifactId>
<version>4.0.0</version>
<name>vripper-core</name>
<description>vripper-core</description>
<properties>
<java.version>17</java.version>
<kotlin.version>1.8.21</kotlin.version>
<maven.deploy.skip>false</maven.deploy.skip>
</properties>
<dependencies>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
</dependency>
<dependency>
<artifactId>kotlin-stdlib</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core-jvm</artifactId>
</dependency>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-java-time</artifactId>
</dependency>
<dependency>
<groupId>io.insert-koin</groupId>
<artifactId>koin-core-jvm</artifactId>
</dependency>
<dependency>
<artifactId>jackson-module-kotlin</artifactId>
<groupId>com.fasterxml.jackson.module</groupId>
</dependency>
<dependency>
<artifactId>spring-boot-starter-jdbc</artifactId>
<groupId>org.springframework.boot</groupId>
<artifactId>h2</artifactId>
<groupId>com.h2database</groupId>
</dependency>
<dependency>
<artifactId>liquibase-core</artifactId>
<groupId>org.liquibase</groupId>
<artifactId>httpclient5</artifactId>
<groupId>org.apache.httpcomponents.client5</groupId>
</dependency>
<dependency>
<artifactId>hsqldb</artifactId>
<groupId>org.hsqldb</groupId>
<scope>runtime</scope>
</dependency>
<dependency>
<artifactId>httpclient</artifactId>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
</dependency>
<dependency>
<artifactId>htmlcleaner</artifactId>
<groupId>net.sourceforge.htmlcleaner</groupId>
<version>2.26</version>
</dependency>
<dependency>
<artifactId>reactor-core</artifactId>
<groupId>io.projectreactor</groupId>
</dependency>
<dependency>
<artifactId>failsafe</artifactId>
<groupId>net.jodah</groupId>
<version>2.4.4</version>
</dependency>
<dependency>
<artifactId>caffeine</artifactId>
@@ -80,16 +82,17 @@
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</dependency>
<dependency>
<artifactId>spring-boot-starter-test</artifactId>
<groupId>org.springframework.boot</groupId>
<scope>test</scope>
<artifactId>commons-codec</artifactId>
<groupId>commons-codec</groupId>
</dependency>
<dependency>
<artifactId>spring-restdocs-mockmvc</artifactId>
<groupId>org.springframework.restdocs</groupId>
<scope>test</scope>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
</dependency>
<dependency>
<artifactId>liquibase-core</artifactId>
<groupId>org.liquibase</groupId>
</dependency>
</dependencies>
@@ -100,29 +103,22 @@
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<artifactId>kotlin-maven-allopen</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<sourceDirs>
<source>src/main/kotlin</source>
<source>target/generated-sources/annotations</source>
</sourceDirs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<distributionManagement>
<repository>
<id>github</id>
<name>GitHub death-claw Apache Maven Packages</name>
<url>https://maven.pkg.github.com/death-claw/vripper-project</url>
</repository>
</distributionManagement>
</project>
@@ -1,25 +1,23 @@
package me.mnlr.vripper
import org.springframework.stereotype.Service
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.download.PostDownloadRunnable
import me.mnlr.vripper.exception.PostParseException
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.repositories.ThreadRepository
import me.mnlr.vripper.services.*
import me.mnlr.vripper.tasks.LinkScanRunnable
import me.mnlr.vripper.tasks.ThreadLookupRunnable
import java.util.*
import java.util.concurrent.CompletableFuture
import java.util.regex.Pattern
@Service
class AppEndpointService(
private val downloadService: DownloadService,
private val dataTransaction: DataTransaction,
private val threadRepository: ThreadRepository,
private val threadCacheService: ThreadCacheService,
private val eventRepository: LogEventRepository,
private val threadPoolService: ThreadPoolService,
private val settingsService: SettingsService
) {
private val log by LoggerDelegate()
@@ -29,9 +27,35 @@ class AppEndpointService(
log.warn("Nothing to scan")
return
}
val urlList = postLinks.split(Pattern.compile("\\r?\\n")).dropLastWhile { it.isEmpty() }.map { it.trim() }
.filter { it.isNotEmpty() }
threadPoolService.generalExecutor.submit(LinkScanRunnable(urlList))
val urlList = postLinks.split(Pattern.compile("\\r?\\n")).dropLastWhile { it.isEmpty() }
.map { it.trim() }.filter { it.isNotEmpty() }
for (link in urlList) {
log.debug("Starting to process thread: $link")
if (!link.startsWith(settingsService.settings.viperSettings.host)) {
continue
}
var threadId: String
var postId: String?
val m = Pattern.compile(
Pattern.quote(settingsService.settings.viperSettings.host) + "/threads/(\\d+)((.*p=)(\\d+))?"
).matcher(link)
if (m.find()) {
threadId = m.group(1)
postId = m.group(4)
if (postId == null) {
CompletableFuture.runAsync(ThreadLookupRunnable(
threadId, settingsService.settings
))
} else {
CompletableFuture.runAsync(PostDownloadRunnable(
threadId, postId
))
}
} else {
log.error("Cannot retrieve thread id from URL $link")
continue
}
}
}
@Synchronized
@@ -41,7 +65,11 @@ class AppEndpointService(
@Synchronized
fun download(posts: List<Pair<String, String>>) {
downloadService.download(posts)
posts.forEach {
CompletableFuture.runAsync(PostDownloadRunnable(
it.first, it.second
))
}
}
@Synchronized
@@ -64,7 +92,7 @@ class AppEndpointService(
@Throws(PostParseException::class)
fun grab(threadId: String): List<PostItem> {
return try {
val thread = threadRepository.findByThreadId(threadId).orElseThrow {
val thread = dataTransaction.findThreadByThreadId(threadId).orElseThrow {
PostParseException(
String.format(
"Unable to find links for threadId = %s", threadId
@@ -0,0 +1,13 @@
package me.mnlr.vripper
import java.nio.file.Files
import java.nio.file.Path
object ApplicationProperties {
const val BASE_DIR_NAME: String = "vripper"
val baseDir: String = System.getProperty("base.dir", System.getProperty("user.dir"))
init {
Files.createDirectories(Path.of(baseDir, BASE_DIR_NAME))
}
}
@@ -1,19 +0,0 @@
package me.mnlr.vripper
import org.springframework.context.event.ContextRefreshedEvent
import org.springframework.context.event.EventListener
import org.springframework.stereotype.Component
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.services.DataTransaction
@Component
class EventListenerBean(
private val postDownloadStateRepository: PostDownloadStateRepository,
private val dataTransaction: DataTransaction
) {
@EventListener
fun onApplicationEvent(event: ContextRefreshedEvent?) {
postDownloadStateRepository.setDownloadingToStopped()
dataTransaction.sortPostsByRank()
}
}
@@ -1,58 +0,0 @@
package me.mnlr.vripper
import jakarta.annotation.PreDestroy
import org.springframework.beans.factory.annotation.Value
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.stereotype.Component
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.services.ThreadPoolService
import java.nio.file.Path
import java.time.format.DateTimeFormatter
import java.util.*
import java.util.regex.Pattern
@Component
class Management(
private val jdbcTemplate: JdbcTemplate,
@Value("\${base.dir}") baseDir: String,
@Value("\${base.dir.name}") baseDirName: String,
private val threadPoolService: ThreadPoolService,
private val downloadService: DownloadService
) {
private val backupFolder: Path
init {
backupFolder = Path.of(baseDir, baseDirName, "backup")
}
@PreDestroy
fun destroy() {
threadPoolService.destroy()
downloadService.destroy()
jdbcTemplate.execute("SHUTDOWN")
}
// @PostConstruct
// @Scheduled(cron = "0 0 0 ? * *")
// private fun backup() {
//
// for (file in Files.list(backupFolder)) {
// val matcher = BACKUP_FILE_PATTERN.matcher(file.fileName.toString())
// if (matcher.find()) {
// val localDate = LocalDate.parse(matcher.group(1), FORMATTER)
// if (localDate.isEqual(LocalDate.now())) {
// return
// }
// }
// }
//
// // create a backup file
// val backupFile = backupFolder.resolve("db_${LocalDateTime.now().format(FORMATTER)}.tar.gz")
// jdbcTemplate.execute("BACKUP DATABASE TO '${backupFile}' BLOCKING")
// }
companion object {
val FORMATTER: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd")
private val BACKUP_FILE_PATTERN = Pattern.compile("^db_(\\d{4}-\\d{2}-\\d{2})\\.tar\\.gz$")
}
}
@@ -0,0 +1,106 @@
package me.mnlr.vripper
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.host.*
import me.mnlr.vripper.repositories.*
import me.mnlr.vripper.repositories.impl.*
import me.mnlr.vripper.services.*
import org.koin.dsl.bind
import org.koin.dsl.module
val coreModule = module {
single<EventBus> {
EventBus
}
single<SettingsService> {
SettingsService(get())
}
single<ImageRepository> {
ImageRepositoryImpl()
}
single<PostDownloadStateRepository> {
PostDownloadStateRepositoryImpl()
}
single<MetadataRepository> {
MetadataRepositoryImpl()
}
single<ThreadRepository> {
ThreadRepositoryImpl()
}
single<LogEventRepository> {
LogEventRepositoryImpl()
}
single<DataTransaction> {
DataTransaction(get(), get(), get(), get(), get(), get())
}
single<RetryPolicyService> {
RetryPolicyService(get(), get())
}
single<HTTPService> {
HTTPService(get(), get())
}
single<VGAuthService> {
VGAuthService(get(), get(), get())
}
single<ThreadCacheService> {
ThreadCacheService(get())
}
single<DownloadService> {
DownloadService(get(), get(), get(), get(), get())
}
single<DownloadSpeedService> {
DownloadSpeedService(get())
}
single<AppEndpointService> {
AppEndpointService(get(), get(), get(), get(), get())
}
single {
AcidimgHost(get(), get(), get())
} bind Host::class
single {
DPicMeHost(get(), get(), get())
} bind Host::class
single {
ImageBamHost(get(), get(), get())
} bind Host::class
single {
ImageTwistHost(get(), get(), get())
} bind Host::class
single {
ImageVenueHost(get(), get(), get())
} bind Host::class
single {
ImageZillaHost(get(), get(), get())
} bind Host::class
single {
ImgboxHost(get(), get(), get())
} bind Host::class
single {
ImgSpiceHost(get(), get(), get())
} bind Host::class
single {
ImxHost(get(), get(), get())
} bind Host::class
single {
PimpandhostHost(get(), get(), get())
} bind Host::class
single {
PixhostHost(get(), get(), get())
} bind Host::class
single {
PixRouteHost(get(), get(), get())
} bind Host::class
single {
PixxxelsHost(get(), get(), get())
} bind Host::class
single {
PostImgHost(get(), get(), get())
} bind Host::class
single {
TurboImageHost(get(), get(), get())
} bind Host::class
single {
ViprImHost(get(), get(), get())
} bind Host::class
}
@@ -1,37 +0,0 @@
package me.mnlr.vripper
import org.springframework.beans.BeansException
import org.springframework.context.ApplicationContext
import org.springframework.context.ApplicationContextAware
import org.springframework.context.ConfigurableApplicationContext
import org.springframework.scheduling.annotation.EnableScheduling
import org.springframework.stereotype.Component
import me.mnlr.vripper.delegate.LoggerDelegate
@Component
@EnableScheduling
class SpringContext : ApplicationContextAware {
@Throws(BeansException::class)
override fun setApplicationContext(context: ApplicationContext) {
// store ApplicationContext reference to access required beans later on
Companion.context = context as ConfigurableApplicationContext
}
companion object {
private val log by LoggerDelegate()
private lateinit var context: ConfigurableApplicationContext
fun <T> getBean(beanClass: Class<T>): T {
return context.getBean(beanClass)
}
fun <T> getBeansOfType(beanClass: Class<T>): Map<String, T> {
return context.getBeansOfType(beanClass)
}
fun close() {
log.info("Application terminating...")
context.close()
}
}
}
@@ -2,6 +2,7 @@ package me.mnlr.vripper
import java.io.PrintWriter
import java.io.StringWriter
import java.security.MessageDigest
fun Throwable.formatToString(): String {
return StringWriter().use { stringWriter ->
@@ -37,3 +38,10 @@ fun getFileNameWithoutExtension(fileName: String): String {
fileName.lastIndexOf(".")
) else fileName
}
fun String.hash256(): String {
val bytes = this.toByteArray()
val md = MessageDigest.getInstance("SHA-256")
val digest = md.digest(bytes)
return digest.fold("") { str, it -> str + "%02x".format(it) }
}
@@ -1,101 +1,84 @@
package me.mnlr.vripper.download
import jakarta.annotation.PostConstruct
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.RetryPolicy
import org.springframework.beans.factory.annotation.Value
import org.springframework.stereotype.Service
import kotlinx.coroutines.*
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.ErrorCountEvent
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.QueueStateEvent
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.repositories.ImageRepository
import me.mnlr.vripper.model.ErrorCount
import me.mnlr.vripper.model.QueueState
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.services.*
import java.util.*
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit
import me.mnlr.vripper.services.DataTransaction
import me.mnlr.vripper.services.RetryPolicyService
import me.mnlr.vripper.services.SettingsService
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.RetryPolicy
import java.util.concurrent.locks.ReentrantLock
import java.util.stream.Collectors
import kotlin.concurrent.withLock
@Service
class DownloadService(
@param:Value("\${download.pool-size}") private val maxPoolSize: Int,
private val settingsService: SettingsService,
private val dataTransaction: DataTransaction,
private val metadataService: MetadataService,
private val hosts: List<Host>,
private val eventRepository: LogEventRepository,
private val retryPolicyService: RetryPolicyService,
private val postDownloadStateRepository: PostDownloadStateRepository,
private val imageRepository: ImageRepository,
private val threadPoolService: ThreadPoolService,
private val eventRepository: LogEventRepository,
private val eventBus: EventBus
) {
private val maxPoolSize: Int = 12
private val log by LoggerDelegate()
// Class fields
private val executor: ExecutorService = Executors.newFixedThreadPool(maxPoolSize)
private val running: MutableMap<Host, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val pending: MutableMap<Host, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val running: MutableMap<String, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val pending: MutableMap<String, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val lock = ReentrantLock()
private val condition = lock.newCondition()
private val pollThread: Thread
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
init {
pollThread = Thread(
{
val accepted: MutableList<ImageDownloadRunnable> = mutableListOf()
val candidates: MutableList<ImageDownloadRunnable> = mutableListOf()
while (!Thread.interrupted()) {
lock.withLock {
candidates.addAll(getCandidates(candidateCount()))
candidates.forEach {
if (canRun(it.context.image.host)) {
accepted.add(it)
running[it.context.image.host]!!.add(it)
log.debug("${it.context.image.url} accepted to run")
}
fun init() {
coroutineScope.launch(Dispatchers.Default) {
val accepted: MutableList<ImageDownloadRunnable> = mutableListOf()
val candidates: MutableList<ImageDownloadRunnable> = mutableListOf()
while (isActive) {
lock.withLock {
candidates.addAll(getCandidates(candidateCount()))
candidates.forEach {
if (canRun(it.context.image.host)) {
accepted.add(it)
running[it.context.image.host]!!.add(it)
log.debug("${it.context.image.url} accepted to run")
}
accepted.forEach {
pending[it.context.image.host]?.remove(it)
schedule(it)
}
accepted.clear()
candidates.clear()
}
accepted.forEach {
pending[it.context.image.host]?.remove(it)
scheduleForDownload(it)
}
accepted.clear()
candidates.clear()
try {
condition.await()
} catch (e: InterruptedException) {
Thread.currentThread().interrupt()
}
}
}, "Download scheduler thread"
)
}
@PostConstruct
fun init() {
pollThread.start()
}
}
}
fun destroy() {
log.info("Shutting down ExecutionService")
pollThread.interrupt()
executor.shutdown()
stop(postDownloadStateRepository.findAll().map { it.postId })
if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {
log.warn("Some jobs are still running!, forcing shutdown")
executor.shutdownNow()
}
stop(dataTransaction.findAllPosts().map { it.postId })
}
fun stopAll(postIds: List<String>?) {
if (postIds != null) {
stop(postIds)
} else {
stop(postDownloadStateRepository.findAll().map(PostDownloadState::postId))
stop(dataTransaction.findAllPosts().map(Post::postId))
}
}
@@ -103,48 +86,46 @@ class DownloadService(
if (postIds.isNotEmpty()) {
restart(postIds)
} else {
restart(postDownloadStateRepository.findAll().map(PostDownloadState::postId))
restart(dataTransaction.findAllPosts().map(Post::postId))
}
}
private fun restart(postIds: List<String>) {
lock.withLock {
val data: MutableMap<PostDownloadState, Collection<ImageDownloadState>> = mutableMapOf()
val data: MutableMap<Post, Collection<Image>> = mutableMapOf()
for (postId in postIds) {
if (isPending(postId)) {
log.warn("Cannot restart, jobs are currently running for post id $postIds")
continue
}
val imageDownloadStates: List<ImageDownloadState> =
imageRepository.findByPostIdAndIsNotCompleted(postId)
if (imageDownloadStates.isEmpty()) {
val images: List<Image> =
dataTransaction.findByPostIdAndIsNotCompleted(postId)
if (images.isEmpty()) {
continue
}
val postDownloadState: PostDownloadState =
postDownloadStateRepository.findByPostId(postId).orElseThrow()
log.debug("Restarting ${imageDownloadStates.size} jobs for post id $postIds")
postDownloadState.status = Status.PENDING
dataTransaction.update(postDownloadState)
data[postDownloadState] = imageDownloadStates
val post: Post =
dataTransaction.findPostsByPostId(postId).orElseThrow()
log.debug("Restarting {} jobs for post id {}", images.size, postIds)
post.status = Status.PENDING
dataTransaction.update(post)
data[post] = images
}
for ((postDownloadState, imageDownloadStates) in data) {
postDownloadState.status = Status.PENDING
dataTransaction.update(postDownloadState)
for (imageDownloadState in imageDownloadStates) {
log.debug("Enqueuing a job for ${imageDownloadState.url}")
with(imageDownloadState) {
this.status = Status.STOPPED
for ((_, images) in data) {
for (image in images) {
log.debug("Enqueuing a job for ${image.url}")
with(image) {
this.status = Status.PENDING
this.current = 0
}
dataTransaction.update(imageDownloadState)
dataTransaction.update(image)
val imageDownloadRunnable = ImageDownloadRunnable(
imageDownloadState.id!!, settingsService.settings
image.id, settingsService.settings
)
pending.computeIfAbsent(
imageDownloadState.host
image.host
) { mutableListOf() }
pending[imageDownloadState.host]!!.add(imageDownloadRunnable)
pending[image.host]!!.add(imageDownloadRunnable)
}
}
condition.signal()
@@ -166,8 +147,8 @@ class DownloadService(
private fun stop(postIds: List<String>) {
lock.withLock {
for (postId in postIds) {
val postDownloadState: PostDownloadState =
postDownloadStateRepository.findByPostId(postId).orElseThrow()
val post: Post =
dataTransaction.findPostsByPostId(postId).orElseThrow()
pending.values.forEach { pending ->
pending.removeIf { it.context.image.postId == postId }
}
@@ -175,32 +156,32 @@ class DownloadService(
.filter { p: ImageDownloadRunnable -> p.context.image.postId == postId }
.forEach { obj: ImageDownloadRunnable -> obj.stop() }
dataTransaction.stopImagesByPostIdAndIsNotCompleted(postId)
dataTransaction.finishPost(postDownloadState)
dataTransaction.finishPost(post)
}
metadataService.stopFetchingMetadata(postIds)
}
}
private fun canRun(host: Host): Boolean {
private fun canRun(host: String): Boolean {
val totalRunning = running.values.sumOf { it.size }
return (running[host]!!.size < settingsService.settings.connectionSettings.maxThreads && if (settingsService.settings.connectionSettings.maxTotalThreads == 0) totalRunning < maxPoolSize else totalRunning < settingsService.settings.connectionSettings.maxTotalThreads)
}
private fun candidateCount(): Map<Host, Int> {
val map: MutableMap<Host, Int> = mutableMapOf()
hosts.forEach { h: Host ->
private fun candidateCount(): Map<String, Int> {
val map: MutableMap<String, Int> = mutableMapOf()
Host.getHosts().forEach { host: String ->
val imageDownloadRunnableList: List<ImageDownloadRunnable> = running.computeIfAbsent(
h
host
) { mutableListOf() }
val count: Int = settingsService.settings.connectionSettings.maxThreads - imageDownloadRunnableList.size
log.debug("Download slots for ${h.host}: $count")
map[h] = count
val count: Int =
settingsService.settings.connectionSettings.maxThreads - imageDownloadRunnableList.size
log.debug("Download slots for $host: $count")
map[host] = count
}
return map
}
private fun getCandidates(candidateCount: Map<Host, Int>): List<ImageDownloadRunnable> {
val hostIntegerMap: MutableMap<Host, Int> = candidateCount.toMutableMap()
private fun getCandidates(candidateCount: Map<String, Int>): List<ImageDownloadRunnable> {
val hostIntegerMap: MutableMap<String, Int> = candidateCount.toMutableMap()
val candidates: MutableList<ImageDownloadRunnable> = mutableListOf()
hosts@ for (host in pending.keys) {
val list: List<ImageDownloadRunnable> =
@@ -217,21 +198,20 @@ class DownloadService(
}
}
if (log.isDebugEnabled) {
val collect: Map<Host, List<ImageDownloadRunnable>> =
val collect: Map<String, List<ImageDownloadRunnable>> =
candidates.stream().collect(Collectors.groupingBy { it.context.image.host })
collect.forEach {
log.debug(
"Candidate download for ${it.key.host} ${it.value.size}/${candidateCount[it.key]}"
"Candidate download for ${it.key} ${it.value.size}/${candidateCount[it.key]}"
)
}
}
return candidates.sortedWith(Comparator.comparing { v: ImageDownloadRunnable -> v.context.post.rank })
}
private fun schedule(imageDownloadRunnable: ImageDownloadRunnable) {
private fun scheduleForDownload(imageDownloadRunnable: ImageDownloadRunnable) {
log.debug("Scheduling a job for ${imageDownloadRunnable.context.image.url}")
executor.execute {
try {
eventBus.publishEvent(QueueStateEvent(QueueState(runningCount(), pendingCount())))
Failsafe.with<Any, RetryPolicy<Any>>(retryPolicyService.buildRetryPolicyForDownload())
.onFailure {
try {
@@ -252,23 +232,29 @@ class DownloadService(
val image = imageDownloadRunnable.context.image
image.status = Status.ERROR
dataTransaction.update(image)
eventBus.publishEvent(ErrorCountEvent(ErrorCount(dataTransaction.countImagesInError())))
}.onComplete {
afterJobFinish(imageDownloadRunnable)
eventBus.publishEvent(
QueueStateEvent(
QueueState(
runningCount(),
pendingCount()
)
)
)
log.debug(
"Finished downloading ${imageDownloadRunnable.context.image.url}"
)
}.run(imageDownloadRunnable::run)
} catch (ignored: Exception) {
}
}
}.runAsync(imageDownloadRunnable::run)
}
fun afterJobFinish(imageDownloadRunnable: ImageDownloadRunnable) {
private fun afterJobFinish(imageDownloadRunnable: ImageDownloadRunnable) {
lock.withLock {
running[imageDownloadRunnable.context.image.host]!!.remove(imageDownloadRunnable)
if (!isPending(imageDownloadRunnable.context.image.postId) && !isRunning(
imageDownloadRunnable.context.image.postId
val image = imageDownloadRunnable.context.image
running[image.host]!!.remove(imageDownloadRunnable)
if (!isPending(image.postId) && !isRunning(
image.postId
)
) {
dataTransaction.finishPost(imageDownloadRunnable.context.post)
@@ -277,21 +263,11 @@ class DownloadService(
}
}
fun pendingCount(): Int {
private fun pendingCount(): Int {
return pending.values.sumOf { it.size }
}
fun runningCount(): Int {
private fun runningCount(): Int {
return running.values.sumOf { it.size }
}
fun download(posts: List<Pair<String, String>>) {
for (post in posts) {
threadPoolService.generalExecutor.submit(
PostDownloadRunnable(
post.first, post.second
)
)
}
}
}
@@ -1,25 +1,22 @@
package me.mnlr.vripper.download
import org.apache.http.client.protocol.HttpClientContext
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.repositories.ImageRepository
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.services.DataTransaction
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
class ImageDownloadContext(val imageId: Long) {
private val imageRepository: ImageRepository =
SpringContext.getBean(ImageRepository::class.java)
private val postDownloadStateRepository: PostDownloadStateRepository =
SpringContext.getBean(PostDownloadStateRepository::class.java)
class ImageDownloadContext(val imageId: Long, val settings: Settings) : KoinComponent {
private val dataTransaction: DataTransaction by inject()
val httpContext: HttpClientContext = HttpClientContext.create()
val postId = image.postIdRef
var stopped = false
val image: ImageDownloadState
get() = imageRepository.findById(imageId).orElseThrow()
val post: PostDownloadState
get() = postDownloadStateRepository.findById(postId).orElseThrow()
val image: Image
get() = dataTransaction.findImageById(imageId).orElseThrow()
val post: Post
get() = dataTransaction.findPostById(postId).orElseThrow()
}
@@ -1,20 +1,22 @@
package me.mnlr.vripper.download
import net.jodah.failsafe.function.CheckedRunnable
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.impl.client.BasicCookieStore
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.getExtension
import me.mnlr.vripper.host.DownloadedImage
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.host.ImageMimeType
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.services.*
import net.jodah.failsafe.function.CheckedRunnable
import org.apache.hc.client5.http.cookie.BasicCookieStore
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.io.IOException
import java.nio.file.Files
import java.nio.file.Path
@@ -24,15 +26,18 @@ import kotlin.io.path.pathString
class ImageDownloadRunnable(
private val imageInternalId: Long, private val settings: Settings
) : CheckedRunnable {
) : KoinComponent, CheckedRunnable {
private val log by LoggerDelegate()
private val dataTransaction: DataTransaction = SpringContext.getBean(DataTransaction::class.java)
private val pathService: PathService = SpringContext.getBean(PathService::class.java)
private val dataTransaction: DataTransaction by inject()
private val hosts: List<Host> = getKoin().getAll()
val context: ImageDownloadContext = ImageDownloadContext(imageInternalId)
private val image: ImageDownloadState
val context: ImageDownloadContext = ImageDownloadContext(imageInternalId, settings)
private val image: Image
get() = context.image
private val post: Post
get() = context.post
private var stopped: Boolean
get() = context.stopped
set(value) {
@@ -57,7 +62,7 @@ class ImageDownloadRunnable(
this.current = 0
}
dataTransaction.update(image)
synchronized(image.postId.intern()) {
synchronized(post.id.toString().intern()) {
val post = context.post
if (post.status != Status.DOWNLOADING) {
post.status = Status.DOWNLOADING
@@ -68,45 +73,48 @@ class ImageDownloadRunnable(
if (stopped) {
return
}
log.debug("Getting image url and name from ${image.url} using ${image.host}")
val downloadedImage = image.host.downloadInternal(image.url, context)
val host = hosts.first { it.isSupported(image.url) }
val downloadedImage = host.downloadInternal(image.url, context)
log.debug("Resolved name for ${image.url}: ${downloadedImage.name}")
log.debug(
"Downloaded image ${image.url} to ${downloadedImage.path}"
)
val sanitizedFileName = pathService.sanitize(downloadedImage.name)
val sanitizedFileName = PathUtils.sanitize(downloadedImage.name)
log.debug(
"Sanitizing image name from ${downloadedImage.name} to $sanitizedFileName"
)
checkImageTypeAndRename(
context.post, downloadedImage, image.index
)
synchronized(image.postId.intern()) {
val post = context.post
post.done += 1
dataTransaction.update(post)
}
} catch (e: Exception) {
if (stopped) {
return
}
throw DownloadException(e)
} finally {
val image = image
if (image.current == image.total && image.total > 0) {
image.status = Status.FINISHED
} else if (stopped) {
image.status = Status.STOPPED
} else {
image.status = Status.ERROR
synchronized(post.id.toString().intern()) {
val image = image
if (image.current == image.total && image.total > 0) {
image.status = Status.FINISHED
val post = context.post
post.done += 1
dataTransaction.update(post)
} else if (stopped) {
image.status = Status.STOPPED
} else {
image.status = Status.ERROR
}
dataTransaction.update(image)
}
dataTransaction.update(image)
}
}
@Throws(HostException::class)
private fun checkImageTypeAndRename(
postDownloadState: PostDownloadState, downloadedImage: DownloadedImage, index: Int
post: Post, downloadedImage: DownloadedImage, index: Int
) {
val existingExtension = getExtension(downloadedImage.name)
val extension = when (downloadedImage.type) {
@@ -116,9 +124,10 @@ class ImageDownloadRunnable(
ImageMimeType.IMAGE_PNG -> "PNG"
ImageMimeType.IMAGE_WEBP -> "WEBP"
}
val filename = if(existingExtension.isBlank()) "${downloadedImage.name}.$extension" else downloadedImage.name
val filename =
if (existingExtension.isBlank()) "${downloadedImage.name}.$extension" else downloadedImage.name
try {
val downloadDestinationFolder = Path.of(postDownloadState.downloadDirectory)
val downloadDestinationFolder = Path.of(post.downloadDirectory)
synchronized(downloadDestinationFolder.pathString.intern()) {
Files.createDirectories(downloadDestinationFolder)
}
@@ -165,7 +174,8 @@ class ImageDownloadRunnable(
fun stop() {
stopped = true
val contextAttributes = httpContext.getAttribute(
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES, HTTPService.ContextAttributes::class.java
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES,
HTTPService.ContextAttributes::class.java
)
if (contextAttributes != null) {
synchronized(contextAttributes.requests) {
@@ -1,14 +1,5 @@
package me.mnlr.vripper.download
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.http.client.HttpClient
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.client.utils.URIBuilder
import org.apache.http.util.EntityUtils
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
@@ -19,24 +10,29 @@ import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.parser.ThreadLookupAPIResponseHandler
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.*
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.net.URIBuilder
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.net.URISyntaxException
import java.util.*
import javax.xml.parsers.SAXParserFactory
class PostDownloadRunnable(private val threadId: String, private val postId: String) : Runnable {
class PostDownloadRunnable(private val threadId: String, private val postId: String) : KoinComponent, Runnable {
private val log by LoggerDelegate()
private val dataTransaction: DataTransaction =
SpringContext.getBean(DataTransaction::class.java)
private val settingsService: SettingsService =
SpringContext.getBean(SettingsService::class.java)
private val vgAuthService: VGAuthService = SpringContext.getBean(VGAuthService::class.java)
private val eventRepository: LogEventRepository =
SpringContext.getBean(LogEventRepository::class.java)
private val cm: HTTPService = SpringContext.getBean(HTTPService::class.java)
private val retryPolicyService: RetryPolicyService =
SpringContext.getBean(RetryPolicyService::class.java)
private val downloadService: DownloadService =
SpringContext.getBean(DownloadService::class.java)
private val dataTransaction: DataTransaction by inject()
private val settingsService: SettingsService by inject()
private val vgAuthService: VGAuthService by inject()
private val eventRepository: LogEventRepository by inject()
private val cm: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val downloadService: DownloadService by inject()
private val link: String =
"${settingsService.settings.viperSettings.host}/threads/$threadId?p=$postId"
private val logEvent: LogEvent
@@ -135,17 +131,17 @@ class PostDownloadRunnable(private val threadId: String, private val postId: Str
"parsing failed for thread $threadId, post $postId", it.failure
)
}.get(CheckedSupplier {
val connection: HttpClient = cm.client.build()
val connection: HttpClient = cm.clientBuilder.build()
(connection.execute(
httpGet, vgAuthService.context
) as CloseableHttpResponse).use { response ->
try {
if (response.statusLine.statusCode / 100 != 2) {
throw DownloadException("Unexpected response code '${response.statusLine.statusCode}' for $httpGet")
if (response.code / 100 != 2) {
throw DownloadException("Unexpected response code '${response.code}' for $httpGet")
}
factory.newSAXParser()
.parse(response.entity.content, threadLookupAPIResponseHandler)
threadLookupAPIResponseHandler.result.postItemList[0]
threadLookupAPIResponseHandler.result.postItemList.first()
} finally {
EntityUtils.consumeQuietly(response.entity)
}
@@ -1,13 +1,13 @@
package me.mnlr.vripper.entities
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.host.Host
data class ImageDownloadState(
var id: Long? = null,
data class Image(
var id: Long = -1,
val postId: String,
val url: String,
val host: Host,
val thumbUrl: String,
val host: String,
val index: Int,
val postIdRef: Long,
var total: Long = -1,
@@ -18,7 +18,7 @@ data class ImageDownloadState(
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as ImageDownloadState
other as Image
if (url != other.url) return false
@@ -1,12 +1,13 @@
package me.mnlr.vripper.entities
import com.fasterxml.jackson.annotation.JsonFormat
import java.time.LocalDateTime
data class LogEvent(
val id: Long? = null,
val type: Type,
val status: Status,
val time: LocalDateTime = LocalDateTime.now(),
@JsonFormat(pattern="yyyy-MM-dd HH:mm:ss") val time: LocalDateTime = LocalDateTime.now(),
val message: String,
) {
@@ -1,10 +1,11 @@
package me.mnlr.vripper.entities
import com.fasterxml.jackson.annotation.JsonFormat
import me.mnlr.vripper.entities.domain.Status
import java.time.LocalDateTime
data class PostDownloadState(
var id: Long? = null,
data class Post(
var id: Long = -1,
val postTitle: String,
val threadTitle: String,
val forum: String,
@@ -15,7 +16,7 @@ data class PostDownloadState(
val total: Int,
val hosts: Set<String>,
val downloadDirectory: String,
val addedOn: LocalDateTime = LocalDateTime.now(),
@JsonFormat(pattern="yyyy-MM-dd HH:mm:ss") val addedOn: LocalDateTime = LocalDateTime.now(),
var status: Status = Status.STOPPED,
var done: Int = 0,
var rank: Int = Int.MAX_VALUE
@@ -24,7 +25,7 @@ data class PostDownloadState(
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as PostDownloadState
other as Post
if (postId != other.postId) return false
@@ -2,6 +2,7 @@ package me.mnlr.vripper.entities
data class Thread(
val id: Long? = null,
val title: String,
val link: String,
val threadId: String,
var total: Int = 0,
@@ -1,7 +1,23 @@
package me.mnlr.vripper.event
data class Event<T>(val kind: Kind, val data: T) {
enum class Kind {
POST_UPDATE, POST_REMOVE, IMAGE_UPDATE, METADATA_UPDATE, THREAD_UPDATE, THREAD_REMOVE, LOG_EVENT_UPDATE, LOG_EVENT_REMOVE, VG_USER, DOWNLOAD_STATUS, BYTES_PER_SECOND, SETTINGS_UPDATE
}
}
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.model.DownloadSpeed
import me.mnlr.vripper.model.ErrorCount
import me.mnlr.vripper.model.QueueState
import me.mnlr.vripper.model.Settings
data class PostCreateEvent(val post: Post)
data class PostUpdateEvent(val post: Post)
data class PostDeleteEvent(val postId: String)
data class ImageCreateEvent(val image: Image)
data class ImageUpdateEvent(val image: Image)
data class ThreadCreateEvent(val thread: Thread)
data class ThreadDeleteEvent(val threadId: String)
class ThreadClearEvent
data class VGUserLoginEvent(val username: String)
data class DownloadSpeedEvent(val downloadSpeed: DownloadSpeed)
data class QueueStateEvent(val queueState: QueueState)
data class ErrorCountEvent(val errorCount: ErrorCount)
data class SettingsUpdateEvent(val settings: Settings)
@@ -1,30 +1,29 @@
package me.mnlr.vripper.event
import jakarta.annotation.PreDestroy
import org.springframework.stereotype.Service
import reactor.core.publisher.Flux
import reactor.core.publisher.SignalType
import reactor.core.publisher.Sinks
import reactor.core.publisher.Sinks.EmitFailureHandler
import reactor.core.publisher.Sinks.EmitResult
import kotlinx.coroutines.channels.BufferOverflow
import kotlinx.coroutines.ensureActive
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.asSharedFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.flow.filterIsInstance
import kotlin.coroutines.coroutineContext
@Service
class EventBus {
private val sink = Sinks.many().multicast().onBackpressureBuffer<Event<*>>()
fun publishEvent(event: Event<*>) {
sink.emitNext(event, RETRY)
object EventBus {
private val _events = MutableSharedFlow<Any>(
onBufferOverflow = BufferOverflow.DROP_OLDEST,
extraBufferCapacity = 100_000
)
val events = _events.asSharedFlow()
fun publishEvent(event: Any) {
_events.tryEmit(event)
}
fun flux(): Flux<Event<*>> {
return sink.asFlux()
suspend inline fun <reified T> subscribe(crossinline onEvent: (T) -> Unit) {
events.filterIsInstance<T>()
.collectLatest { event ->
coroutineContext.ensureActive()
onEvent(event)
}
}
@PreDestroy
private fun destroy() {
sink.emitComplete(RETRY)
}
companion object {
val RETRY = EmitFailureHandler { _: SignalType?, _: EmitResult? -> true }
}
}
}
@@ -1,52 +1,42 @@
package me.mnlr.vripper.host
import org.apache.http.NameValuePair
import org.apache.http.client.HttpClient
import org.apache.http.client.entity.UrlEncodedFormEntity
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.message.BasicNameValuePair
import org.apache.http.util.EntityUtils
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class AcidimgHost(
private val httpService: HTTPService,
private val htmlProcessorService: HtmlProcessorService,
private val xpathService: XpathService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("acidimg.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
url: String, document: Document, context: ImageDownloadContext
): Pair<String, String> {
try {
log.debug(
String.format(
"Looking for xpath expression %s in %s",
CONTINUE_BUTTON_XPATH,
url
"Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url
)
)
xpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
} catch (e: XpathException) {
throw HostException(e)
}
log.debug(String.format("Click button found for %s", url))
val client: HttpClient = httpService.client.build()
val client: HttpClient = httpService.clientBuilder.build()
val httpPost = httpService.buildHttpPost(url, context.httpContext)
httpPost.addHeader("Referer", url)
val params: MutableList<NameValuePair> = ArrayList()
@@ -63,7 +53,7 @@ class AcidimgHost(
) as CloseableHttpResponse).use { response ->
log.debug(String.format("Cleaning response for %s", httpPost))
try {
htmlProcessorService.clean(response.entity.content)
HtmlProcessorService.clean(response.entity.content)
} finally {
EntityUtils.consumeQuietly(response.entity)
}
@@ -73,14 +63,12 @@ class AcidimgHost(
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(doc, IMG_XPATH)
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
"Xpath '%s' cannot be found in '%s'", IMG_XPATH, url
)
)
return try {
@@ -96,8 +84,6 @@ class AcidimgHost(
}
companion object {
private const val host = "acidimg.cc"
private const val lookup = "acidimg.cc"
private const val CONTINUE_BUTTON_XPATH = "//input[@id='continuebutton']"
private const val IMG_XPATH = "//img[@class='centred']"
}
@@ -1,26 +1,20 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
@Service
class DPicMeHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("dpic.me", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -30,7 +24,7 @@ class DPicMeHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -62,8 +56,6 @@ class DPicMeHost(
}
companion object {
private const val host = "dpic.me"
private const val lookup = "dpic.me"
private const val IMG_XPATH = "//img[@id='pic']"
}
}
@@ -1,25 +1,24 @@
package me.mnlr.vripper.host
import org.apache.http.Header
import org.apache.http.client.HttpClient
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.util.EntityUtils
import org.w3c.dom.Document
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.getFileNameWithoutExtension
import me.mnlr.vripper.services.*
import java.io.FileOutputStream
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.Header
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.w3c.dom.Document
import java.nio.file.Files
import java.nio.file.Path
import kotlin.io.path.pathString
abstract class Host(
val hostId: String,
private val httpService: HTTPService,
private val htmlProcessorService: HtmlProcessorService,
private val dataTransaction: DataTransaction,
private val downloadSpeedService: DownloadSpeedService
) {
@@ -27,9 +26,16 @@ abstract class Host(
companion object {
private const val READ_BUFFER_SIZE = 8192
private val hosts: MutableList<String> = mutableListOf()
fun getHosts(): List<String> {
return hosts.toList()
}
}
abstract val host: String
init {
hosts.add(hostId)
}
@Throws(HostException::class)
abstract fun resolve(
@@ -56,7 +62,7 @@ abstract class Host(
if (value != null) {
if (value.contains("text/html")) {
val document = fetch(url, context) {
htmlProcessorService.clean(it.entity.content)
HtmlProcessorService.clean(it.entity.content)
}
if (log.isDebugEnabled) {
log.debug("Cleaning $url response", url)
@@ -81,13 +87,13 @@ abstract class Host(
response: CloseableHttpResponse,
context: ImageDownloadContext
): Pair<Path, ImageMimeType> {
val mimeType = getImageMimeType(response.allHeaders)
val mimeType = getImageMimeType(response.headers)
?: throw HostException("Unsupported image type ${response.getFirstHeader("content-type")}")
val tempImage = Files.createTempFile("vripper", "tmp")
val tempImage = Files.createTempFile(Path.of(context.settings.downloadSettings.tempPath), "vripper_", ".tmp")
return response.entity.content.use { inputStream ->
try {
FileOutputStream(tempImage.pathString).use { fos ->
Files.newOutputStream(tempImage).use { fos ->
val image = context.image
image.total = response.entity.contentLength
dataTransaction.update(image)
@@ -106,7 +112,7 @@ abstract class Host(
with(image) {
current += read
}
downloadSpeedService.increase(read.toLong())
downloadSpeedService.reportDownloadedBytes(read.toLong())
dataTransaction.update(image)
}
Pair(tempImage, mimeType)
@@ -118,12 +124,12 @@ abstract class Host(
}
fun isSupported(url: String): Boolean {
return url.contains(host)
return url.contains(hostId)
}
@Throws(HostException::class)
fun head(url: String, context: HttpClientContext): Array<Header> {
val client: HttpClient = httpService.client.build()
val client: CloseableHttpAsyncClient = httpService.clientBuilder.build()
val httpGet = httpService.buildHttpHead(url, context)
log.debug(String.format("Requesting %s", url))
return try {
@@ -132,14 +138,14 @@ abstract class Host(
context
) as CloseableHttpResponse).use { response ->
try {
if (response.statusLine.statusCode / 100 != 2) {
if (response.code / 100 != 2) {
throw HostException(
String.format(
"Unexpected response code: %d", response.statusLine.statusCode
"Unexpected response code: %d", response.code
)
)
}
response.allHeaders
response.headers
} finally {
EntityUtils.consumeQuietly(response.entity)
}
@@ -155,7 +161,7 @@ abstract class Host(
context: ImageDownloadContext,
transformer: (CloseableHttpResponse) -> T
): T {
val client: HttpClient = httpService.client.build()
val client: HttpClient = httpService.clientBuilder.build()
val httpGet = httpService.buildHttpGet(url, context.httpContext)
httpGet.addHeader("Referer", context.image.url)
log.debug(String.format("Requesting %s", url))
@@ -164,10 +170,10 @@ abstract class Host(
httpGet,
context.httpContext
) as CloseableHttpResponse).use { response ->
if (response.statusLine.statusCode / 100 != 2) {
if (response.code / 100 != 2) {
EntityUtils.consumeQuietly(response.entity)
throw DownloadException(
"Server returned code ${response.statusLine.statusCode}"
"Server returned code ${response.code}"
)
}
try {
@@ -208,13 +214,17 @@ abstract class Host(
other as Host
if (host != other.host) return false
if (hostId != other.hostId) return false
return true
}
override fun hashCode(): Int {
return host.hashCode()
return hostId.hashCode()
}
override fun toString(): String {
return hostId
}
}
@@ -1,30 +1,24 @@
package me.mnlr.vripper.host
import org.apache.http.impl.cookie.BasicClientCookie
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.impl.cookie.BasicClientCookie
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.sql.Date
import java.time.LocalDateTime
import java.time.ZoneId
import java.util.*
@Service
class ImageBamHost(
private val htmlProcessorService: HtmlProcessorService,
private val xpathService: XpathService,
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imagebam.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -34,7 +28,7 @@ class ImageBamHost(
): Pair<String, String> {
val doc = try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_XPATH, url))
if (xpathService.getAsNode(document, CONTINUE_XPATH) != null) {
if (XpathService.getAsNode(document, CONTINUE_XPATH) != null) {
val clientCookie = BasicClientCookie("nsfw_inter", "1")
clientCookie.domain = "www.imagebam.com"
clientCookie.path = "/"
@@ -44,7 +38,7 @@ class ImageBamHost(
)
context.httpContext.cookieStore.addCookie(clientCookie)
fetch(url, context) {
htmlProcessorService.clean(it.entity.content)
HtmlProcessorService.clean(it.entity.content)
}
} else {
document
@@ -54,7 +48,7 @@ class ImageBamHost(
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(doc, IMG_XPATH)
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -84,8 +78,6 @@ class ImageBamHost(
}
companion object {
private const val host = "imagebam.com"
private const val lookup = "imagebam.com"
private const val IMG_XPATH = "//img[contains(@class,'main-image')]"
private const val CONTINUE_XPATH = "//*[contains(text(), 'Continue')]"
}
@@ -1,26 +1,20 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
@Service
class ImageTwistHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imagetwist.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -30,7 +24,7 @@ class ImageTwistHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -54,8 +48,6 @@ class ImageTwistHost(
}
companion object {
private const val host = "imagetwist.com"
private const val lookup = "imagetwist.com"
private const val IMG_XPATH = "//img[contains(@class, 'img')]"
}
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class ImageVenueHost(
private val htmlProcessorService: HtmlProcessorService,
private val xpathService: XpathService,
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imagevenue.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -35,10 +29,10 @@ class ImageVenueHost(
url
)
)
if (xpathService.getAsNode(document, CONTINUE_BUTTON_XPATH) != null) {
if (XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH) != null) {
// Button detected. No need to actually click it, just make the call again.
fetch(url, context) {
htmlProcessorService.clean(it.entity.content)
HtmlProcessorService.clean(it.entity.content)
}
} else {
document
@@ -48,7 +42,7 @@ class ImageVenueHost(
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(doc, IMG_XPATH)
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -72,8 +66,6 @@ class ImageVenueHost(
}
companion object {
private const val host = "imagevenue.com"
private const val lookup = "imagevenue.com"
private const val CONTINUE_BUTTON_XPATH = "//a[@title='Continue to ImageVenue']"
private const val IMG_XPATH = "//a[@data-toggle='full']/img"
}
@@ -1,24 +1,18 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
@Service
class ImageZillaHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imagezilla.net", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -28,7 +22,7 @@ class ImageZillaHost(
): Pair<String, String> {
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -52,8 +46,6 @@ class ImageZillaHost(
}
companion object {
private const val host = "imagezilla.net"
private const val lookup = "imagezilla.net/show"
private const val IMG_XPATH = "//img[@id='photo']"
}
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class ImgSpiceHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imgspice.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -29,7 +23,7 @@ class ImgSpiceHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -52,8 +46,6 @@ class ImgSpiceHost(
}
companion object {
private const val host = "imgspice.com"
private const val lookup = "imgspice.com"
private const val IMG_XPATH = "//img[@id='imgpreview']"
}
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class ImgboxHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imgbox.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -29,7 +23,7 @@ class ImgboxHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -50,8 +44,6 @@ class ImgboxHost(
}
companion object {
private const val host = "imgbox.com"
private const val lookup = "imgbox.com"
private const val IMG_XPATH = "//img[@id='img']"
}
}
@@ -1,34 +1,28 @@
package me.mnlr.vripper.host
import org.apache.http.NameValuePair
import org.apache.http.client.HttpClient
import org.apache.http.client.entity.UrlEncodedFormEntity
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.methods.HttpPost
import org.apache.http.message.BasicNameValuePair
import org.apache.http.util.EntityUtils
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.HtmlProcessorException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.io.IOException
@Service
class ImxHost(
private val httpService: HTTPService,
private val htmlProcessorService: HtmlProcessorService,
private val xpathService: XpathService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("imx.to", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -36,10 +30,11 @@ class ImxHost(
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val httpsUrl = url.replace("http:", "https:")
var value: String? = null
try {
log.debug("Looking for xpath expression $CONTINUE_BUTTON_XPATH in $url")
val contDiv = xpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
log.debug("Looking for xpath expression $CONTINUE_BUTTON_XPATH in $httpsUrl")
val contDiv = XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
?: throw HostException("$CONTINUE_BUTTON_XPATH cannot be found")
val node = contDiv.attributes.getNamedItem("value")
if (node != null) {
@@ -51,9 +46,9 @@ class ImxHost(
if (value == null) {
throw HostException("Failed to obtain value attribute from continue input")
}
log.debug("Click button found for $url")
val client: HttpClient = httpService.client.build()
val httpPost: HttpPost = httpService.buildHttpPost(url, context.httpContext)
log.debug("Click button found for $httpsUrl")
val client: HttpClient = httpService.clientBuilder.build()
val httpPost: HttpPost = httpService.buildHttpPost(httpsUrl, context.httpContext)
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("imgContinue", value))
try {
@@ -68,7 +63,7 @@ class ImxHost(
) as CloseableHttpResponse).use { response ->
log.debug("Cleaning response for $httpPost")
try {
htmlProcessorService.clean(response.entity.content)
HtmlProcessorService.clean(response.entity.content)
} finally {
EntityUtils.consumeQuietly(response.entity)
}
@@ -79,15 +74,15 @@ class ImxHost(
throw HostException(e)
}
val imgNode: Node = try {
log.debug("Looking for xpath expression $IMG_XPATH in $url")
xpathService.getAsNode(doc, IMG_XPATH)
log.debug("Looking for xpath expression $IMG_XPATH in $httpsUrl")
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
"Xpath $IMG_XPATH cannot be found in $url"
"Xpath $IMG_XPATH cannot be found in $httpsUrl"
)
return try {
log.debug("Resolving name and image url for $url")
log.debug("Resolving name and image url for $httpsUrl")
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(
@@ -99,8 +94,6 @@ class ImxHost(
}
companion object {
private const val host = "imx.to"
private const val lookup = "imx.to"
private const val CONTINUE_BUTTON_XPATH = "//*[@name='imgContinue']"
private const val IMG_XPATH = "//img[@class='centred']"
}
@@ -1,27 +1,21 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.net.URI
import java.net.URISyntaxException
@Service
class PimpandhostHost(
private val xpathService: XpathService,
private val htmlProcessorService: HtmlProcessorService,
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("pimpandhost.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -31,16 +25,16 @@ class PimpandhostHost(
): Pair<String, String> {
val newUrl: String
try {
newUrl = appendUri(url.replace("http://", "https://"), "size=original")
newUrl = appendUri(url.replace("http://", "https://").replace("-medium(\\.html)?".toRegex(), ""), "size=original")
} catch (e: Exception) {
throw HostException(e)
}
val doc = fetch(newUrl, context) {
htmlProcessorService.clean(it.entity.content)
HtmlProcessorService.clean(it.entity.content)
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, newUrl))
xpathService.getAsNode(doc, IMG_XPATH)
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -78,8 +72,6 @@ class PimpandhostHost(
}
companion object {
private const val host = "pimpandhost.com"
private const val lookup = "pimpandhost.com"
private const val IMG_XPATH = "//img[contains(@class, 'original')]"
}
}
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class PixRouteHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("pixroute.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -29,7 +23,7 @@ class PixRouteHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -49,8 +43,6 @@ class PixRouteHost(
}
companion object {
private const val host = "pixroute.com"
private const val lookup = "pixroute.com"
private const val IMG_XPATH = "//img[@id='imgpreview']"
}
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class PixhostHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("pixhost.to", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -29,7 +23,7 @@ class PixhostHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -50,8 +44,6 @@ class PixhostHost(
}
companion object {
private const val host = "pixhost.to"
private const val lookup = "pixhost.to"
private const val IMG_XPATH = "//img[@id='image']"
}
}
@@ -1,24 +1,18 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
@Service
class PixxxelsHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("pixxxels.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -28,7 +22,7 @@ class PixxxelsHost(
): Pair<String, String> {
val imgNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -40,7 +34,7 @@ class PixxxelsHost(
)
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
xpathService.getAsNode(document, TITLE_XPATH)
XpathService.getAsNode(document, TITLE_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -63,8 +57,6 @@ class PixxxelsHost(
}
companion object {
private const val host = "pixxxels.cc"
private const val lookup = "pixxxels.cc"
private const val IMG_XPATH = "//*[@id='download']"
private const val TITLE_XPATH = "//*[contains(@class,'imagename')]"
}
@@ -1,26 +1,20 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
@Service
class PostImgHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("postimg.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -30,7 +24,7 @@ class PostImgHost(
): Pair<String, String> {
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
xpathService.getAsNode(document, TITLE_XPATH)
XpathService.getAsNode(document, TITLE_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -42,7 +36,7 @@ class PostImgHost(
)
val urlNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -64,8 +58,6 @@ class PostImgHost(
}
companion object {
private const val host = "postimg.cc"
private const val lookup = "postimg.cc"
private const val TITLE_XPATH = "//span[contains(@class,'imagename')]"
private const val IMG_XPATH = "//a[@id='download']"
}
@@ -1,25 +1,19 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
@Service
class TurboImageHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("turboimagehost.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String
get() = Companion.host
@Throws(HostException::class)
override fun resolve(
@@ -30,7 +24,7 @@ class TurboImageHost(
var title: String?
title = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
val titleNode: Node? = xpathService.getAsNode(document, TITLE_XPATH)
val titleNode: Node? = XpathService.getAsNode(document, TITLE_XPATH)
log.debug(String.format("Resolving name for %s", url))
titleNode?.textContent?.trim { it <= ' ' }
} catch (e: XpathException) {
@@ -39,7 +33,7 @@ class TurboImageHost(
if (title.isNullOrEmpty()) {
title = getDefaultImageName(url)
}
val urlNode: Node = xpathService.getAsNode(document, IMG_XPATH)
val urlNode: Node = XpathService.getAsNode(document, IMG_XPATH)
?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
@@ -51,8 +45,6 @@ class TurboImageHost(
}
companion object {
private const val host = "turboimagehost.com"
private const val lookup = "turboimagehost.com"
private const val TITLE_XPATH = "//div[contains(@class,'titleFullS')]/h1"
private const val IMG_XPATH = "//img[@id='imageid']"
}
@@ -1,25 +1,20 @@
package me.mnlr.vripper.host
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
@Service
class ViprImHost(
private val xpathService: XpathService,
httpService: HTTPService,
htmlProcessorService: HtmlProcessorService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host(httpService, htmlProcessorService, dataTransaction, downloadSpeedService) {
) : Host("vipr.im", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
override val host: String = "vipr.im"
@Throws(HostException::class)
override fun resolve(
@@ -29,7 +24,7 @@ class ViprImHost(
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
xpathService.getAsNode(document, IMG_XPATH)
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
@@ -0,0 +1,32 @@
package me.mnlr.vripper.listeners
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import java.io.RandomAccessFile
import java.nio.file.Files
import kotlin.io.path.Path
import kotlin.io.path.pathString
import kotlin.system.exitProcess
object AppLock {
fun exclusiveLock() {
val lock = Path(baseDir).resolve(BASE_DIR_NAME).resolve("lock")
try {
val randomFile = RandomAccessFile(lock.pathString, "rw")
val channel = randomFile.channel
val fileLock = channel.tryLock()
if (fileLock == null) {
System.err.println("Another instance is already running in ${lock.parent.pathString}")
exitProcess(-1)
} else {
Runtime.getRuntime().addShutdownHook(Thread {
fileLock.release()
channel.close()
Files.deleteIfExists(lock)
})
}
} catch (e: Exception) {
println(e.toString())
}
}
}
@@ -0,0 +1,25 @@
package me.mnlr.vripper.listeners
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.services.*
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
open class OnStartupListener : KoinComponent {
private val dataTransaction: DataTransaction by inject()
private val downloadService: DownloadService by inject()
private val httpService: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val vgAuthService: VGAuthService by inject()
private val downloadSpeedService: DownloadSpeedService by inject()
open fun run() {
dataTransaction.setDownloadingToStopped()
dataTransaction.sortPostsByRank()
httpService.init()
retryPolicyService.init()
vgAuthService.init()
downloadService.init()
downloadSpeedService.init()
}
}
@@ -1,19 +1,3 @@
package me.mnlr.vripper.model
data class DownloadSpeed(val speed: String) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as DownloadSpeed
if (speed != other.speed) return false
return true
}
override fun hashCode(): Int {
return speed.hashCode()
}
}
data class DownloadSpeed(val speed: String)
@@ -0,0 +1,3 @@
package me.mnlr.vripper.model
data class ErrorCount(val count: Int)
@@ -1,33 +0,0 @@
package me.mnlr.vripper.model
data class GlobalState(
val running: Int,
val remaining: Int,
val error: Int,
val loggedUser: String,
val downloadSpeed: String
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as GlobalState
if (running != other.running) return false
if (remaining != other.remaining) return false
if (error != other.error) return false
if (loggedUser != other.loggedUser) return false
if (downloadSpeed != other.downloadSpeed) return false
return true
}
override fun hashCode(): Int {
var result = running
result = 31 * result + remaining
result = 31 * result + error
result = 31 * result + loggedUser.hashCode()
result = 31 * result + downloadSpeed.hashCode()
return result
}
}
@@ -1,7 +1,5 @@
package me.mnlr.vripper.model
import me.mnlr.vripper.host.Host
data class PostItem(
val threadId: String,
val threadTitle: String,
@@ -10,8 +8,8 @@ data class PostItem(
val title: String,
val imageCount: Int,
val url: String,
val hosts: Map<Host, Int>,
val hosts: List<Pair<String, Int>>,
val securityToken: String,
val forum: String,
val imageItemList: List<ImageItem>
)
)
@@ -0,0 +1,6 @@
package me.mnlr.vripper.model
data class QueueState(
val running: Int,
val remaining: Int
)
@@ -1,12 +1,12 @@
package me.mnlr.vripper.model
class Settings {
var desktopClipboard: Boolean = false
var maxEventLog: Int = 1_000
var connectionSettings: ConnectionSettings = ConnectionSettings()
var downloadSettings: DownloadSettings = DownloadSettings()
var viperSettings: ViperSettings = ViperSettings()
}
data class Settings(
var maxEventLog: Int = 1_000,
var connectionSettings: ConnectionSettings = ConnectionSettings(),
var downloadSettings: DownloadSettings = DownloadSettings(),
var viperSettings: ViperSettings = ViperSettings(),
var clipboardSettings: ClipboardSettings = ClipboardSettings()
)
data class ViperSettings(
var login: Boolean = false,
@@ -18,6 +18,7 @@ data class ViperSettings(
data class DownloadSettings(
var downloadPath: String = System.getProperty("user.home"),
var tempPath: String = System.getProperty("java.io.tmpdir"),
var autoStart: Boolean = true,
var autoQueueThreshold: Int = 1,
var forceOrder: Boolean = false,
@@ -28,8 +29,10 @@ data class DownloadSettings(
)
data class ConnectionSettings(
var maxThreads: Int = 4,
var maxThreads: Int = 2,
var maxTotalThreads: Int = 0,
var timeout: Int = 30,
var timeout: Long = 30,
var maxAttempts: Int = 3,
)
)
data class ClipboardSettings(var enable: Boolean = false, var pollingRate: Int = 500)
@@ -1,3 +0,0 @@
package me.mnlr.vripper.model
data class ThreadInput(val link: String, val threadId: String)
@@ -1,14 +1,5 @@
package me.mnlr.vripper.parser
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.http.client.HttpClient
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.client.utils.URIBuilder
import org.apache.http.util.EntityUtils
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.PostParseException
@@ -17,19 +8,27 @@ import me.mnlr.vripper.services.HTTPService
import me.mnlr.vripper.services.RetryPolicyService
import me.mnlr.vripper.services.SettingsService
import me.mnlr.vripper.services.VGAuthService
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.net.URIBuilder
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.io.BufferedInputStream
import java.net.URISyntaxException
import java.util.*
import javax.xml.parsers.SAXParserFactory
class ThreadLookupAPIParser(private val threadId: String) {
class ThreadLookupAPIParser(private val threadId: String) : KoinComponent {
private val log by LoggerDelegate()
private val cm: HTTPService = SpringContext.getBean(HTTPService::class.java)
private val retryPolicyService: RetryPolicyService =
SpringContext.getBean(RetryPolicyService::class.java)
private val vgAuthService: VGAuthService = SpringContext.getBean(VGAuthService::class.java)
private val settingsService: SettingsService =
SpringContext.getBean(SettingsService::class.java)
private val cm: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val vgAuthService: VGAuthService by inject()
private val settingsService: SettingsService by inject()
@Throws(PostParseException::class)
fun parse(): ThreadItem {
@@ -50,13 +49,13 @@ class ThreadLookupAPIParser(private val threadId: String) {
it.failure
)
}.get(CheckedSupplier {
val connection: HttpClient = cm.client.build()
val connection: HttpClient = cm.clientBuilder.build()
(connection.execute(
httpGet, vgAuthService.context
) as CloseableHttpResponse).use { response ->
try {
if (response.statusLine.statusCode / 100 != 2) {
throw DownloadException("Unexpected response code '${response.statusLine.statusCode}' for $httpGet")
if (response.code / 100 != 2) {
throw DownloadException("Unexpected response code '${response.code}' for $httpGet")
}
factory.newSAXParser().parse(
BufferedInputStream(response.entity.content),
@@ -1,21 +1,20 @@
package me.mnlr.vripper.parser
import org.xml.sax.Attributes
import org.xml.sax.helpers.DefaultHandler
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.model.ImageItem
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.services.SettingsService
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import org.xml.sax.Attributes
import org.xml.sax.helpers.DefaultHandler
class ThreadLookupAPIResponseHandler : DefaultHandler() {
class ThreadLookupAPIResponseHandler : KoinComponent, DefaultHandler() {
private val log by LoggerDelegate()
private val supportedHosts: Collection<Host> =
SpringContext.getBeansOfType(Host::class.java).values
private val settingsService: SettingsService =
SpringContext.getBean(SettingsService::class.java)
private val supportedHosts: List<Host> = getKoin().getAll()
private val settingsService: SettingsService by inject()
private var error: String = ""
private val hostMap: MutableMap<Host, Int> = mutableMapOf()
private val postItemList: MutableList<PostItem> = mutableListOf()
@@ -56,7 +55,7 @@ class ThreadLookupAPIResponseHandler : DefaultHandler() {
val mainLink = attributes.getValue("main_url")?.trim() ?: ""
val thumbLink = attributes.getValue("thumb_url")?.trim() ?: ""
val type = attributes.getValue("type")?.trim() ?: ""
if(type == "linked") {
if (type == "linked") {
try {
val host = supportedHosts.first {
it.isSupported(mainLink)
@@ -67,7 +66,7 @@ class ThreadLookupAPIResponseHandler : DefaultHandler() {
}
imageItemList.add(ImageItem(mainLink, thumbLink, host))
} catch (e: Exception) {
log.warn("Unsupported link: $mainLink")
log.warn("Unsupported link: $mainLink", e)
}
}
}
@@ -86,7 +85,7 @@ class ThreadLookupAPIResponseHandler : DefaultHandler() {
postTitle,
imageItemList.size,
"${settingsService.settings.viperSettings.host}/threads/?p=$postId&viewfull=1#post$postId",
hostMap.toMap(),
hostMap.toMap().map { Pair(it.key.hostId, it.value) },
securityToken,
forum,
imageItemList.toList()
@@ -1,16 +1,17 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.entities.Image
import java.util.*
interface ImageRepository {
fun save(imageDownloadState: ImageDownloadState): ImageDownloadState
fun save(image: Image): Image
fun save(imageList: List<Image>)
fun deleteAllByPostId(postId: String)
fun findByPostId(postId: String): List<ImageDownloadState>
fun findByPostId(postId: String): List<Image>
fun countError(): Int
fun findByPostIdAndIsNotCompleted(postId: String): List<ImageDownloadState>
fun findByPostIdAndIsNotCompleted(postId: String): List<Image>
fun stopByPostIdAndIsNotCompleted(postId: String): Int
fun findByPostIdAndIsError(postId: String): List<ImageDownloadState>
fun findById(id: Long): Optional<ImageDownloadState>
fun update(imageDownloadState: ImageDownloadState)
fun findByPostIdAndIsError(postId: String): List<Image>
fun findById(id: Long): Optional<Image>
fun update(image: Image)
}
@@ -1,16 +1,17 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Post
import java.util.*
interface PostDownloadStateRepository {
fun save(postDownloadState: PostDownloadState): PostDownloadState
fun findByPostId(postId: String): Optional<PostDownloadState>
fun findById(id: Long): Optional<PostDownloadState>
fun save(post: Post): Post
fun findByPostId(postId: String): Optional<Post>
fun findById(id: Long): Optional<Post>
fun findCompleted(): List<String>
fun findAll(): List<PostDownloadState>
fun findAll(): List<Post>
fun existByPostId(postId: String): Boolean
fun setDownloadingToStopped(): Int
fun deleteByPostId(postId: String): Int
fun update(postDownloadState: PostDownloadState)
fun update(post: Post)
fun update(post: List<Post>)
}
@@ -1,125 +1,120 @@
package me.mnlr.vripper.repositories.impl
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.jdbc.core.RowMapper
import org.springframework.stereotype.Service
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.repositories.ImageRepository
import java.sql.ResultSet
import java.sql.SQLException
import me.mnlr.vripper.tables.ImageTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
@Service
class ImageRepositoryImpl(
private val jdbcTemplate: JdbcTemplate, private val eventBus: EventBus
) : ImageRepository {
@Synchronized
private fun nextId(): Long {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_IMAGE", Long::class.java)!!
class ImageRepositoryImpl : ImageRepository {
override fun save(image: Image): Image {
val id = ImageTable.insertAndGetId {
it[current] = image.current
it[host] = image.host
it[index] = image.index
it[postId] = image.postId
it[status] = image.status.name
it[total] = image.total
it[url] = image.url
it[thumbUrl] = image.thumbUrl
it[postIdRef] = image.postIdRef
}.value
return image.copy(id = id)
}
override fun save(imageDownloadState: ImageDownloadState): ImageDownloadState {
val id = nextId()
jdbcTemplate.update(
"INSERT INTO IMAGE (ID, CURRENT, HOST, INDEX, POST_ID, STATUS, TOTAL, URL, POST_ID_REF) VALUES (?,?,?,?,?,?,?,?,?)",
id,
imageDownloadState.current,
imageDownloadState.host.host,
imageDownloadState.index,
imageDownloadState.postId,
imageDownloadState.status.name,
imageDownloadState.total,
imageDownloadState.url,
imageDownloadState.postIdRef
)
imageDownloadState.id = id
eventBus.publishEvent(Event(Event.Kind.IMAGE_UPDATE, id))
return imageDownloadState
}
override fun deleteAllByPostId(postId: String) {
jdbcTemplate.update("DELETE FROM IMAGE WHERE POST_ID = ?", postId)
}
override fun findByPostId(postId: String): List<ImageDownloadState> {
return jdbcTemplate.query(
"SELECT * FROM IMAGE WHERE POST_ID = ?", ImageRowMapper(), postId
)
}
override fun countError(): Int {
return jdbcTemplate.queryForObject(
"SELECT COUNT(*) FROM IMAGE AS image WHERE image.STATUS = ?", Int::class.java, Status.ERROR.name
)
}
override fun findByPostIdAndIsNotCompleted(postId: String): List<ImageDownloadState> {
return jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.POST_ID = ? AND image.STATUS <> ?",
ImageRowMapper(),
postId,
Status.FINISHED.name
)
}
override fun stopByPostIdAndIsNotCompleted(postId: String): Int {
return jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.STATUS = ? WHERE image.POST_ID = ? AND image.STATUS <> ?",
Status.STOPPED.name,
postId,
Status.FINISHED.name
)
}
override fun findByPostIdAndIsError(postId: String): List<ImageDownloadState> {
return jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.POST_ID = ? AND image.STATUS = ?",
ImageRowMapper(),
postId,
Status.ERROR.name
)
}
override fun findById(id: Long): Optional<ImageDownloadState> {
val images = jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.ID = ?", ImageRowMapper(), id
)
return if (images.isEmpty()) {
Optional.empty()
} else {
Optional.of(images[0])
override fun save(imageList: List<Image>) {
ImageTable.batchInsert(imageList, shouldReturnGeneratedValues = false) {
this[ImageTable.current] = it.current
this[ImageTable.host] = it.host
this[ImageTable.index] = it.index
this[ImageTable.postId] = it.postId
this[ImageTable.status] = it.status.name
this[ImageTable.total] = it.total
this[ImageTable.url] = it.url
this[ImageTable.thumbUrl] = it.thumbUrl
this[ImageTable.postIdRef] = it.postIdRef
}
}
override fun update(imageDownloadState: ImageDownloadState) {
jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.STATUS = ?, image.CURRENT = ?, image.TOTAL = ? WHERE image.ID = ?",
imageDownloadState.status.name,
imageDownloadState.current,
imageDownloadState.total,
imageDownloadState.id
override fun deleteAllByPostId(postId: String) {
ImageTable.deleteWhere { ImageTable.postId eq postId }
}
override fun findByPostId(postId: String): List<Image> {
return ImageTable.select {
ImageTable.postId eq postId
}.map(this::transform)
}
override fun countError(): Int {
return ImageTable
.slice(ImageTable.id)
.select { ImageTable.status eq Status.ERROR.name }
.count().toInt()
}
override fun findByPostIdAndIsNotCompleted(postId: String): List<Image> {
return ImageTable
.select {
(ImageTable.postId eq postId) and (ImageTable.status neq Status.FINISHED.name)
}.map(this::transform)
}
override fun stopByPostIdAndIsNotCompleted(postId: String): Int {
return ImageTable.update({ (ImageTable.postId eq postId) and (ImageTable.status neq Status.FINISHED.name) }) {
it[status] = Status.STOPPED.name
}
}
override fun findByPostIdAndIsError(postId: String): List<Image> {
return ImageTable.select {
(ImageTable.postId eq postId) and (ImageTable.status eq Status.ERROR.name)
}.map(this::transform)
}
override fun findById(id: Long): Optional<Image> {
val result = ImageTable.select {
ImageTable.id eq id
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun update(image: Image) {
ImageTable.update({ ImageTable.id eq image.id }) {
it[status] = image.status.name
it[current] = image.current
it[total] = image.total
}
}
private fun transform(resultRow: ResultRow): Image {
val id = resultRow[ImageTable.id].value
val postId = resultRow[ImageTable.postId]
val url = resultRow[ImageTable.url]
val thumbUrl = resultRow[ImageTable.thumbUrl]
val host = resultRow[ImageTable.host]
val index = resultRow[ImageTable.index]
val current = resultRow[ImageTable.current]
val total = resultRow[ImageTable.total]
val status = Status.valueOf(resultRow[ImageTable.status])
val postIdRef = resultRow[ImageTable.postIdRef]
return Image(
id,
postId,
url,
thumbUrl,
host,
index,
postIdRef,
total,
current,
status
)
eventBus.publishEvent(Event(Event.Kind.IMAGE_UPDATE, imageDownloadState.id))
}
}
internal class ImageRowMapper : RowMapper<ImageDownloadState> {
@Throws(SQLException::class)
override fun mapRow(rs: ResultSet, rowNum: Int): ImageDownloadState {
val id = rs.getLong("ID")
val postId = rs.getString("POST_ID")
val url = rs.getString("URL")
val host = SpringContext.getBeansOfType(Host::class.java).values.filter { it.host == rs.getString("HOST") }[0]
val index = rs.getInt("INDEX")
val current = rs.getLong("CURRENT")
val total = rs.getLong("TOTAL")
val status = Status.valueOf(rs.getString("STATUS"))
val postIdRef = rs.getLong("POST_ID_REF")
return ImageDownloadState(id, postId, url, host, index, postIdRef, total, current, status)
}
}
@@ -1,107 +1,33 @@
package me.mnlr.vripper.repositories.impl
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.jdbc.core.RowMapper
import org.springframework.stereotype.Service
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.SettingsService
import java.sql.ResultSet
import java.sql.SQLException
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter
import java.util.*
@Service
class LogEventRepositoryImpl(
private val jdbcTemplate: JdbcTemplate,
private val settingsService: SettingsService,
private val eventBus: EventBus
) : LogEventRepository {
private val log by LoggerDelegate()
@Synchronized
private fun nextId(): Long {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_EVENT", Long::class.java)!!
}
class LogEventRepositoryImpl : LogEventRepository {
@Synchronized
override fun save(logEvent: LogEvent): LogEvent {
val maxRecords = settingsService.settings.maxEventLog - 1
val count = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM EVENT", Long::class.java) ?: 0
if (count > maxRecords) {
val idList = jdbcTemplate.queryForList(
"SELECT ID FROM EVENT ORDER BY TIME ASC LIMIT ?",
Long::class.java,
count - maxRecords
)
idList.forEach { delete(it) }
}
val id = nextId()
jdbcTemplate.update(
"INSERT INTO EVENT (ID, TYPE, STATUS, TIME, MESSAGE) VALUES (?,?,?,?,?)",
id,
logEvent.type.name,
logEvent.status.name,
logEvent.time.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME),
logEvent.message
)
eventBus.publishEvent(Event(Event.Kind.LOG_EVENT_UPDATE, id))
return logEvent.copy(id = id)
return logEvent
}
override fun update(logEvent: LogEvent): LogEvent {
if (logEvent.id == null) {
log.warn("Cannot update entity with null id")
return logEvent
}
jdbcTemplate.update(
"UPDATE EVENT SET STATUS = ?, MESSAGE = ? WHERE ID = ?",
logEvent.status.name,
logEvent.message,
logEvent.id
)
eventBus.publishEvent(Event(Event.Kind.LOG_EVENT_UPDATE, logEvent.id))
return logEvent
}
override fun findById(id: Long): Optional<LogEvent> {
val logEvents =
jdbcTemplate.query("SELECT * FROM EVENT WHERE ID = ?", LogEventRowMapper(), id)
return if (logEvents.isEmpty()) {
Optional.empty()
} else {
Optional.of(logEvents[0])
}
return Optional.empty()
}
override fun findAll(): List<LogEvent> {
return jdbcTemplate.query("SELECT * FROM EVENT", LogEventRowMapper())
return emptyList()
}
override fun delete(id: Long) {
jdbcTemplate.update("DELETE FROM EVENT WHERE ID = ?", id)
eventBus.publishEvent(Event(Event.Kind.LOG_EVENT_REMOVE, id))
}
override fun deleteAll() {
jdbcTemplate.update("DELETE FROM EVENT")
}
}
internal class LogEventRowMapper : RowMapper<LogEvent> {
@Throws(SQLException::class)
override fun mapRow(rs: ResultSet, rowNum: Int): LogEvent {
val id = rs.getLong("ID")
val type = LogEvent.Type.valueOf(rs.getString("TYPE"))
val status = LogEvent.Status.valueOf(rs.getString("STATUS"))
val time =
LocalDateTime.parse(rs.getString("TIME"), DateTimeFormatter.ISO_LOCAL_DATE_TIME)
val message = rs.getString("MESSAGE")
return LogEvent(id, type, status, time, message)
}
}
@@ -1,65 +1,20 @@
package me.mnlr.vripper.repositories.impl
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.jdbc.core.RowMapper
import org.springframework.stereotype.Service
import me.mnlr.vripper.entities.Metadata
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.repositories.*
import java.sql.ResultSet
import java.sql.SQLException
import me.mnlr.vripper.repositories.MetadataRepository
import java.util.*
@Service
class MetadataRepositoryImpl(
private val jdbcTemplate: JdbcTemplate,
private val eventBus: EventBus
) : MetadataRepository {
class MetadataRepositoryImpl: MetadataRepository {
override fun save(metadata: Metadata): Metadata {
jdbcTemplate.update(
"INSERT INTO METADATA (POST_ID_REF, POST_ID, POSTED_BY, RESOLVED_NAMES) VALUES (?,?,?,?)",
metadata.postIdRef,
metadata.postId,
metadata.postedBy,
java.lang.String.join("%sep%", metadata.resolvedNames)
)
eventBus.publishEvent(Event(Event.Kind.METADATA_UPDATE, metadata.postIdRef))
return metadata
}
override fun findByPostId(postId: String): Optional<Metadata> {
val metadata = jdbcTemplate.query(
"SELECT metadata.* FROM METADATA AS metadata WHERE metadata.POST_ID = ?",
MetadataRowMapper(),
postId
)
return if (metadata.isEmpty()) {
Optional.empty()
} else {
Optional.of(metadata[0])
}
return Optional.empty()
}
override fun deleteByPostId(postId: String): Int {
return jdbcTemplate.update(
"DELETE FROM METADATA AS metadata WHERE metadata.POST_ID = ?", postId
)
return 0
}
}
internal class MetadataRowMapper : RowMapper<Metadata> {
@Throws(SQLException::class)
override fun mapRow(rs: ResultSet, rowNum: Int): Metadata {
val metadata = Metadata()
metadata.postIdRef = rs.getLong("POST_ID_REF")
metadata.postId = rs.getString("POST_ID")
metadata.postedBy = rs.getString("POSTED_BY")
val resolvedNames = rs.getString("RESOLVED_NAMES")
if (resolvedNames != null && resolvedNames.isNotBlank()) {
metadata.resolvedNames =
resolvedNames.split("%sep%").dropLastWhile { it.isEmpty() }
}
return metadata
}
}
@@ -1,163 +1,145 @@
package me.mnlr.vripper.repositories.impl
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.jdbc.core.RowMapper
import org.springframework.stereotype.Service
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import java.sql.ResultSet
import java.sql.SQLException
import java.sql.Timestamp
import me.mnlr.vripper.tables.PostTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
@Service
class PostDownloadStateRepositoryImpl(
private val jdbcTemplate: JdbcTemplate, private val eventBus: EventBus
) : PostDownloadStateRepository {
@Synchronized
private fun nextId(): Long {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_POST", Long::class.java)!!
class PostDownloadStateRepositoryImpl :
PostDownloadStateRepository {
private val delimiter = ";"
override fun save(post: Post): Post {
val id = PostTable.insertAndGetId {
it[done] = post.done
it[hosts] = java.lang.String.join(delimiter, post.hosts)
it[outputPath] = post.downloadDirectory
it[postId] = post.postId
it[status] = post.status.name
it[threadId] = post.threadId
it[postTitle] = post.postTitle
it[threadTitle] = post.threadTitle
it[forum] = post.forum
it[total] = post.total
it[url] = post.url
it[token] = post.token
it[addedAt] = post.addedOn
it[rank] = post.rank
}.value
return post.copy(id = id)
}
override fun save(postDownloadState: PostDownloadState): PostDownloadState {
val id = nextId()
jdbcTemplate.update(
"INSERT INTO POST (ID, DONE, HOSTS, POST_FOLDER_NAME, POST_ID, STATUS, THREAD_ID, POST_TITLE, THREAD_TITLE, FORUM, TOTAL, URL, TOKEN, ADDED_ON, RANK) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
id,
postDownloadState.done,
java.lang.String.join(";", postDownloadState.hosts),
postDownloadState.downloadDirectory,
postDownloadState.postId,
postDownloadState.status.name,
postDownloadState.threadId,
postDownloadState.postTitle,
postDownloadState.threadTitle,
postDownloadState.forum,
postDownloadState.total,
postDownloadState.url,
postDownloadState.token,
Timestamp.valueOf(postDownloadState.addedOn),
postDownloadState.rank
)
eventBus.publishEvent(Event(Event.Kind.POST_UPDATE, id))
return postDownloadState.copy(id = id)
}
override fun findByPostId(postId: String): Optional<PostDownloadState> {
val post = jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID WHERE post.POST_ID = ?",
PostRowMapper(),
postId
)
return if (post.isEmpty()) {
override fun findByPostId(postId: String): Optional<Post> {
val result = PostTable.select {
PostTable.postId eq postId
}.map { transform(it) }
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(post[0])
Optional.of(result.first())
}
}
override fun findCompleted(): List<String> {
return jdbcTemplate.query(
"SELECT POST_ID FROM POST AS post WHERE status = ? AND done >= total",
{ rs: ResultSet, _: Int -> rs.getString("POST_ID") },
Status.FINISHED.name
)
return PostTable.slice(PostTable.postId).select {
(PostTable.status eq Status.FINISHED.name) and (PostTable.done greaterEq PostTable.total)
}.map { it[PostTable.postId] }
}
override fun findById(id: Long): Optional<PostDownloadState> {
val post = jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID WHERE post.ID = ?",
PostRowMapper(),
id
)
return if (post.isEmpty()) {
override fun findById(id: Long): Optional<Post> {
val result = PostTable.select {
PostTable.id eq id
}.map { transform(it) }
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(post[0])
Optional.of(result.first())
}
}
override fun findAll(): List<PostDownloadState> {
return jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID",
PostRowMapper()
)
override fun findAll(): List<Post> {
return PostTable.selectAll().map { transform(it) }
}
override fun existByPostId(postId: String): Boolean {
val count = jdbcTemplate.queryForObject(
"SELECT COUNT(*) FROM POST AS post WHERE post.POST_ID = ?", Int::class.java, postId
)
return count > 0
return PostTable.slice(PostTable.id).select { PostTable.postId eq postId }.count() > 0
}
override fun setDownloadingToStopped(): Int {
return jdbcTemplate.update(
"UPDATE POST AS post SET post.STATUS = ? WHERE post.STATUS = ? OR post.STATUS = ?",
Status.STOPPED.name,
Status.DOWNLOADING.name,
Status.PENDING.name
)
return PostTable.update({ (PostTable.status eq Status.DOWNLOADING.name) or (PostTable.status eq Status.PENDING.name) }) {
it[status] = Status.STOPPED.name
}
}
override fun deleteByPostId(postId: String): Int {
val mutationCount = jdbcTemplate.update("DELETE FROM POST AS post WHERE post.POST_ID = ?", postId)
eventBus.publishEvent(Event(Event.Kind.POST_REMOVE, postId))
return mutationCount
return PostTable.deleteWhere { PostTable.postId eq postId }
}
override fun update(postDownloadState: PostDownloadState) {
jdbcTemplate.update(
"UPDATE POST AS post SET post.STATUS = ?, post.DONE = ?, post.RANK = ? WHERE post.ID = ?",
postDownloadState.status.name,
postDownloadState.done,
postDownloadState.rank,
postDownloadState.id
override fun update(post: Post) {
PostTable.update({ PostTable.id eq post.id }) {
it[status] = post.status.name
it[done] = post.done
it[rank] = post.rank
}
}
override fun update(post: List<Post>) {
PostTable.batchReplace(post, shouldReturnGeneratedValues = false) {
this[PostTable.id] = it.id
this[PostTable.status] = it.status.name
this[PostTable.done] = it.done
this[PostTable.total] = it.total
this[PostTable.rank] = it.rank
this[PostTable.hosts] = java.lang.String.join(delimiter, it.hosts)
this[PostTable.outputPath] = it.downloadDirectory
this[PostTable.postId] = it.postId
this[PostTable.threadId] = it.threadId
this[PostTable.postTitle] = it.postTitle
this[PostTable.threadTitle] = it.threadTitle
this[PostTable.forum] = it.forum
this[PostTable.url] = it.url
this[PostTable.token] = it.token
}
}
private fun transform(resultRow: ResultRow): Post {
val id = resultRow[PostTable.id].value
val status = Status.valueOf(resultRow[PostTable.status])
val postId = resultRow[PostTable.postId]
val threadId = resultRow[PostTable.threadId]
val postTitle = resultRow[PostTable.postTitle]
val threadTitle = resultRow[PostTable.threadTitle]
val forum = resultRow[PostTable.forum]
val url = resultRow[PostTable.url]
val token = resultRow[PostTable.token]
val done = resultRow[PostTable.done]
val total = resultRow[PostTable.total]
val hosts =
resultRow[PostTable.hosts].split(delimiter).dropLastWhile { it.isEmpty() }.toSet()
val downloadDirectory = resultRow[PostTable.outputPath]
val addedOn = resultRow[PostTable.addedAt]
val rank = resultRow[PostTable.rank]
return Post(
id,
postTitle,
threadTitle,
forum,
url,
token,
postId,
threadId,
total,
hosts,
downloadDirectory,
addedOn,
status,
done,
rank
)
eventBus.publishEvent(Event(Event.Kind.POST_UPDATE, postDownloadState.id))
}
}
internal class PostRowMapper : RowMapper<PostDownloadState> {
@Throws(SQLException::class)
override fun mapRow(rs: ResultSet, rowNum: Int): PostDownloadState {
val id = rs.getLong("post.ID")
val status = Status.valueOf(rs.getString("post.STATUS"))
val postId = rs.getString("post.POST_ID")
val threadId = rs.getString("post.THREAD_ID")
val postTitle = rs.getString("post.POST_TITLE")
val threadTitle = rs.getString("post.THREAD_TITLE")
val forum = rs.getString("post.FORUM")
val url = rs.getString("post.URL")
val token = rs.getString("post.TOKEN")
val done = rs.getInt("post.DONE")
val total = rs.getInt("post.TOTAL")
val hosts = rs.getString("post.HOSTS").split(DELIMITER).dropLastWhile { it.isEmpty() }.toSet()
val downloadDirectory = rs.getString("post.POST_FOLDER_NAME")
val addedOn = rs.getTimestamp("post.ADDED_ON").toLocalDateTime()
val rank = rs.getInt("post.RANK")
// val metadataId = rs.getLong("metadata.POST_ID_REF")
// if (!rs.wasNull()) {
// val metadata = Metadata()
// metadata.postIdRef = metadataId
// metadata.postId = rs.getString("metadata.POST_ID")
// val resolvedNames = rs.getString("metadata.RESOLVED_NAMES")
// if (resolvedNames != null && resolvedNames.isNotBlank()) {
// metadata.resolvedNames =
// resolvedNames.split("%sep%").dropLastWhile { it.isEmpty() }
// }
// metadata.postedBy = rs.getString("metadata.POSTED_BY")
// postDownloadState.metadata = metadata
// }
return PostDownloadState(
id, postTitle, threadTitle, forum, url, token, postId, threadId, total, hosts, downloadDirectory, addedOn, status, done, rank
)
}
companion object {
private const val DELIMITER = ";"
}
}
@@ -1,87 +1,64 @@
package me.mnlr.vripper.repositories.impl
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.jdbc.core.RowMapper
import org.springframework.stereotype.Service
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.repositories.ThreadRepository
import java.sql.ResultSet
import java.sql.SQLException
import me.mnlr.vripper.tables.ThreadTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
@Service
class ThreadRepositoryImpl @Autowired constructor(
private val jdbcTemplate: JdbcTemplate,
private val eventBus: EventBus
) : ThreadRepository {
@Synchronized
private fun nextId(): Long {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_THREAD", Long::class.java)!!
}
class ThreadRepositoryImpl : ThreadRepository {
override fun save(thread: Thread): Thread {
val id = nextId()
jdbcTemplate.update(
"INSERT INTO THREAD (ID, TOTAL, LINK, THREAD_ID) values (?,?,?,?)",
id,
thread.total,
thread.link,
thread.threadId
)
eventBus.publishEvent(Event(Event.Kind.THREAD_UPDATE, id))
val id = ThreadTable.insertAndGetId {
it[title] = thread.title
it[total] = thread.total
it[url] = thread.link
it[threadId] = thread.threadId
}.value
return thread.copy(id = id)
}
override fun findByThreadId(threadId: String): Optional<Thread> {
val threadList = jdbcTemplate.query(
"SELECT * FROM THREAD AS thread WHERE thread.THREAD_ID = ?",
ThreadRowMapper(),
threadId
)
return if (threadList.isEmpty()) {
val result = ThreadTable.select {
ThreadTable.threadId eq threadId
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(threadList[0])
Optional.of(result.first())
}
}
override fun findAll(): List<Thread> {
return jdbcTemplate.query("SELECT * FROM THREAD", ThreadRowMapper())
return ThreadTable.selectAll().map(this::transform)
}
override fun findById(id: Long): Optional<Thread> {
val threadList = jdbcTemplate.query(
"SELECT * FROM THREAD AS thread WHERE thread.ID = ?", ThreadRowMapper(), id
)
return if (threadList.isEmpty()) {
val result = ThreadTable.select {
ThreadTable.id eq id
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(threadList[0])
Optional.of(result.first())
}
}
override fun deleteByThreadId(threadId: String): Int {
val mutationCount =
jdbcTemplate.update("DELETE FROM THREAD AS thread WHERE THREAD_ID = ?", threadId)
eventBus.publishEvent(Event(Event.Kind.THREAD_REMOVE, threadId))
return mutationCount
return ThreadTable.deleteWhere { ThreadTable.threadId eq threadId }
}
override fun deleteAll() {
jdbcTemplate.update("DELETE FROM THREAD")
ThreadTable.deleteAll()
}
private fun transform(resultRow: ResultRow): Thread {
val id = resultRow[ThreadTable.id].value
val title = resultRow[ThreadTable.title]
val url = resultRow[ThreadTable.url]
val threadId = resultRow[ThreadTable.threadId]
val total = resultRow[ThreadTable.total]
return Thread(id, title, url, threadId, total)
}
}
internal class ThreadRowMapper : RowMapper<Thread> {
@Throws(SQLException::class)
override fun mapRow(rs: ResultSet, rowNum: Int): Thread {
val id = rs.getLong("ID")
val link = rs.getString("LINK")
val threadId = rs.getString("THREAD_ID")
val total = rs.getInt("TOTAL")
return Thread(id, link, threadId, total)
}
}
@@ -0,0 +1,5 @@
package me.mnlr.vripper.services
object AppVersion {
const val VERSION: String = "4.4.0"
}
@@ -1,114 +1,172 @@
package me.mnlr.vripper.services
import org.springframework.stereotype.Service
import org.springframework.transaction.annotation.Transactional
import me.mnlr.vripper.entities.ImageDownloadState
import com.github.benmanes.caffeine.cache.Caffeine
import com.github.benmanes.caffeine.cache.LoadingCache
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Metadata
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.*
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.repositories.ImageRepository
import me.mnlr.vripper.repositories.MetadataRepository
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.repositories.ThreadRepository
import org.jetbrains.exposed.sql.transactions.transaction
import java.util.*
import kotlin.io.path.pathString
@Service
@Transactional
class DataTransaction(
private val settingsService: SettingsService,
private val postDownloadStateRepository: PostDownloadStateRepository,
private val imageRepository: ImageRepository,
private val threadRepository: ThreadRepository,
private val metadataRepository: MetadataRepository,
private val settingsService: SettingsService,
private val pathService: PathService
private val eventBus: EventBus,
) {
private fun save(postDownloadState: PostDownloadState): PostDownloadState {
return postDownloadStateRepository.save(postDownloadState)
private val imageIdCache: LoadingCache<Long, Optional<Image>> = Caffeine.newBuilder().build {
_findImageById(it)
}
private val postIdCache: LoadingCache<Long, Optional<Post>> = Caffeine.newBuilder().build {
_findPostById(it)
}
fun update(postDownloadState: PostDownloadState) {
postDownloadStateRepository.update(postDownloadState)
private fun save(post: Post): Post {
return transaction { postDownloadStateRepository.save(post) }
}
fun save(thread: Thread): Thread {
return threadRepository.save(thread)
fun update(post: Post) {
transaction { postDownloadStateRepository.update(post) }
postIdCache.invalidate(post.id)
eventBus.publishEvent(PostUpdateEvent(post))
}
private fun save(imageDownloadState: ImageDownloadState): ImageDownloadState {
return imageRepository.save(imageDownloadState)
fun save(thread: Thread) {
val savedThread = transaction { threadRepository.save(thread) }
eventBus.publishEvent(
ThreadCreateEvent(
savedThread
)
)
}
fun update(imageDownloadState: ImageDownloadState) {
imageRepository.update(imageDownloadState)
fun update(image: Image) {
transaction { imageRepository.update(image) }
imageIdCache.invalidate(image.id)
eventBus.publishEvent(ImageUpdateEvent(image))
}
fun exists(postId: String): Boolean {
return postDownloadStateRepository.existByPostId(postId)
return transaction { postDownloadStateRepository.existByPostId(postId) }
}
fun newPost(postItem: PostItem): PostDownloadState {
val postDownloadState = save(PostDownloadState(
postTitle = postItem.title,
url = postItem.url,
token = postItem.securityToken,
postId = postItem.postId,
threadId = postItem.threadId,
total = postItem.imageCount,
hosts = postItem.hosts.keys.map { it.host }.toSet(),
threadTitle = postItem.threadTitle,
forum = postItem.forum,
downloadDirectory = pathService.calculateDownloadPath(postItem.forum, postItem.threadTitle, postItem.title, postItem.postId, settingsService.settings).pathString
))
val images: MutableList<ImageDownloadState> = mutableListOf()
postItem.imageItemList.forEachIndexed { index, imageItem ->
val imageDownloadState = ImageDownloadState(
postId = postItem.postId,
url = imageItem.mainLink,
host = imageItem.host,
index = index,
postIdRef = postDownloadState.id!!
fun newPost(postItem: PostItem): Post {
val (post, images) = transaction {
val post = save(
Post(
postTitle = postItem.title,
url = postItem.url,
token = postItem.securityToken,
postId = postItem.postId,
threadId = postItem.threadId,
total = postItem.imageCount,
hosts = postItem.hosts.map { "${it.first} (${it.second})" }.toSet(),
threadTitle = postItem.threadTitle,
forum = postItem.forum,
downloadDirectory = PathUtils.calculateDownloadPath(
postItem.forum,
postItem.threadTitle,
postItem.title,
postItem.postId,
settingsService.settings
).pathString
)
)
images.add(save(imageDownloadState))
val images = postItem.imageItemList.mapIndexed { index, imageItem ->
Image(
postId = postItem.postId,
url = imageItem.mainLink,
thumbUrl = imageItem.thumbLink,
host = imageItem.host.hostId,
index = index,
postIdRef = post.id!!
)
}
save(images)
sortPostsByRank()
Pair(post, images)
}
sortPostsByRank()
return postDownloadState
eventBus.publishEvent(PostCreateEvent(post))
images.forEach {
eventBus.publishEvent(ImageCreateEvent(it))
}
return post
}
fun finishPost(postDownloadState: PostDownloadState) {
if (imageRepository.findByPostIdAndIsError(postDownloadState.postId).isNotEmpty()) {
postDownloadState.status = Status.ERROR
update(postDownloadState)
private fun save(images: List<Image>) {
transaction { imageRepository.save(images) }
}
fun finishPost(post: Post) {
val imagesInErrorStatus = findByPostIdAndIsError(post.postId)
if (imagesInErrorStatus.isNotEmpty()) {
post.status = Status.ERROR
update(post)
} else {
if (postDownloadState.done < postDownloadState.total) {
postDownloadState.status = Status.STOPPED
update(postDownloadState)
if (post.done < post.total) {
post.status = Status.STOPPED
update(post)
} else {
postDownloadState.status = Status.FINISHED
update(postDownloadState)
if (settingsService.settings.downloadSettings.clearCompleted) {
remove(listOf(postDownloadState.postId))
post.status = Status.FINISHED
transaction {
update(post)
if (settingsService.settings.downloadSettings.clearCompleted) {
remove(listOf(post.postId))
}
}
}
}
}
private fun findByPostIdAndIsError(postId: String): List<Image> {
return transaction { imageRepository.findByPostIdAndIsError(postId) }
}
private fun remove(postIds: List<String>) {
for (postId in postIds) {
imageRepository.deleteAllByPostId(postId)
metadataRepository.deleteByPostId(postId)
postDownloadStateRepository.deleteByPostId(postId)
transaction {
for (postId in postIds) {
imageRepository.deleteAllByPostId(postId)
metadataRepository.deleteByPostId(postId)
postDownloadStateRepository.deleteByPostId(postId)
}
sortPostsByRank()
}
postIds.forEach {
eventBus.publishEvent(PostDeleteEvent(it))
}
sortPostsByRank()
}
fun removeThread(threadId: String) {
threadRepository.deleteByThreadId(threadId)
transaction { threadRepository.deleteByThreadId(threadId) }
eventBus.publishEvent(ThreadDeleteEvent(threadId))
}
fun clearCompleted(): List<String> {
val completed = postDownloadStateRepository.findCompleted()
val completed = transaction { postDownloadStateRepository.findCompleted() }
remove(completed)
return completed
}
@@ -117,33 +175,98 @@ class DataTransaction(
if (postIds != null) {
remove(postIds)
} else {
remove(postDownloadStateRepository.findAll().map(PostDownloadState::postId))
remove(findAllPosts().map(Post::postId))
}
}
fun stopImagesByPostIdAndIsNotCompleted(postId: String) {
imageRepository.stopByPostIdAndIsNotCompleted(postId)
transaction { imageRepository.stopByPostIdAndIsNotCompleted(postId) }
}
@Synchronized
fun setMetadata(postDownloadState: PostDownloadState, metadata: Metadata) {
if (metadataRepository.findByPostId(postDownloadState.postId).isEmpty) {
metadata.postIdRef = postDownloadState.id
fun setMetadata(post: Post, metadata: Metadata) {
if (metadataRepository.findByPostId(post.postId).isEmpty) {
metadata.postIdRef = post.id
metadataRepository.save(metadata)
}
}
fun clearQueueLinks() {
threadRepository.deleteAll()
transaction { threadRepository.deleteAll() }
eventBus.publishEvent(ThreadClearEvent())
}
@Synchronized
fun sortPostsByRank() {
val postDownloadState =
postDownloadStateRepository.findAll().sortedWith(Comparator.comparing(PostDownloadState::addedOn))
for (i in postDownloadState.indices) {
postDownloadState[i].rank = i
update(postDownloadState[i])
val post =
findAllPosts().sortedWith(Comparator.comparing(Post::addedOn))
for (i in post.indices) {
post[i].rank = i
}
update(post)
}
private fun update(postList: List<Post>) {
transaction { postDownloadStateRepository.update(postList) }
postList.forEach { postIdCache.invalidate(it.id) }
postList.forEach {
eventBus.publishEvent(PostUpdateEvent(it))
}
}
fun setDownloadingToStopped() {
transaction { postDownloadStateRepository.setDownloadingToStopped() }
}
fun findAllPosts(): List<Post> {
return transaction { postDownloadStateRepository.findAll() }
}
fun findPostById(id: Long): Optional<Post> {
return postIdCache[id]
}
private fun _findPostById(id: Long): Optional<Post> {
return transaction { postDownloadStateRepository.findById(id) }
}
fun findImagesByPostId(postId: String): List<Image> {
return transaction { imageRepository.findByPostId(postId) }
}
fun findImageById(id: Long): Optional<Image> {
return imageIdCache[id]
}
private fun _findImageById(id: Long): Optional<Image> {
return transaction { imageRepository.findById(id) }
}
fun findAllThreads(): List<Thread> {
return transaction { threadRepository.findAll() }
}
fun findThreadById(id: Long): Optional<Thread> {
return transaction {
threadRepository.findById(id)
}
}
fun findByPostIdAndIsNotCompleted(postId: String): List<Image> {
return transaction { imageRepository.findByPostIdAndIsNotCompleted(postId) }
}
fun countImagesInError(): Int {
return transaction { imageRepository.countError() }
}
fun findPostsByPostId(postId: String): Optional<Post> {
return transaction { postDownloadStateRepository.findByPostId(postId) }
}
fun findThreadByThreadId(threadId: String): Optional<Thread> {
return transaction { threadRepository.findByThreadId(threadId) }
}
}
@@ -0,0 +1,27 @@
package me.mnlr.vripper.services
import liquibase.Contexts
import liquibase.LabelExpression
import liquibase.Liquibase
import liquibase.database.Database
import liquibase.database.DatabaseFactory
import liquibase.database.jvm.JdbcConnection
import liquibase.resource.ClassLoaderResourceAccessor
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import java.sql.DriverManager
object DatabaseMigration {
fun update() {
val database: Database = DatabaseFactory.getInstance()
.findCorrectDatabaseImplementation(JdbcConnection(DriverManager.getConnection("jdbc:h2:file:$baseDir/$BASE_DIR_NAME/vripper;DB_CLOSE_DELAY=-1;")))
Liquibase(
"db.changelog-master.xml",
ClassLoaderResourceAccessor(),
database
).use { liquibase ->
liquibase.update(Contexts(), LabelExpression())
}
}
}
@@ -1,30 +1,53 @@
package me.mnlr.vripper.services
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Service
import me.mnlr.vripper.event.Event
import kotlinx.coroutines.*
import me.mnlr.vripper.event.DownloadSpeedEvent
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.QueueStateEvent
import me.mnlr.vripper.formatSI
import me.mnlr.vripper.model.DownloadSpeed
import java.util.concurrent.atomic.AtomicLong
@Service
class DownloadSpeedService(private val eventBus: EventBus) {
private val read = AtomicLong(0)
var currentValue = 0L
private var allowWrite = false
fun increase(read: Long) {
if (allowWrite) {
this.read.addAndGet(read)
class DownloadSpeedService(
private val eventBus: EventBus,
) {
companion object {
const val DOWNLOAD_POLL_RATE = 2500
}
private val coroutineScope = CoroutineScope(Dispatchers.Default)
private val bytesCount = AtomicLong(0)
private var job: Job? = null
fun init() {
coroutineScope.launch {
eventBus.subscribe<QueueStateEvent> {
if (it.queueState.running + it.queueState.remaining > 0) {
if (job == null || job?.isActive == false) {
job = coroutineScope.launch {
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(0L.formatSI())))
while (isActive) {
delay(DOWNLOAD_POLL_RATE.toLong())
val newValue = bytesCount.getAndSet(0)
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(((newValue * 1000) / DOWNLOAD_POLL_RATE).formatSI())))
}
}
}
} else {
job?.cancel()
coroutineScope.launch {
delay(DOWNLOAD_POLL_RATE + 500L)
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(0L.formatSI())))
}
}
}
}
}
@Scheduled(fixedDelay = 1000)
private fun calc() {
allowWrite = false
val newValue = read.getAndSet(0)
if (newValue != currentValue) {
currentValue = newValue
eventBus.publishEvent(Event(Event.Kind.BYTES_PER_SECOND, currentValue))
}
allowWrite = true
fun reportDownloadedBytes(count: Long) {
bytesCount.addAndGet(count)
}
}
}
@@ -1,44 +0,0 @@
package me.mnlr.vripper.services
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Service
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.formatSI
import me.mnlr.vripper.model.GlobalState
import me.mnlr.vripper.repositories.ImageRepository
@Service
class GlobalStateService(
private val downloadService: DownloadService,
private val imageRepository: ImageRepository,
private val eventBus: EventBus,
private val vgAuthService: VGAuthService,
private val downloadSpeedService: DownloadSpeedService
) {
private var currentState: GlobalState = newValue()
@Scheduled(fixedDelay = 200)
private fun interval() {
val newGlobalState = newValue()
if (newGlobalState != currentState) {
currentState = newGlobalState
eventBus.publishEvent(Event(Event.Kind.DOWNLOAD_STATUS, currentState))
}
}
private fun newValue(): GlobalState {
return GlobalState(
downloadService.runningCount(),
downloadService.pendingCount(),
imageRepository.countError(),
vgAuthService.loggedUser,
downloadSpeedService.currentValue.formatSI()
)
}
fun get(): GlobalState {
return currentState.copy()
}
}
@@ -1,79 +1,102 @@
package me.mnlr.vripper.services
import jakarta.annotation.PreDestroy
import org.apache.http.client.config.CookieSpecs
import org.apache.http.client.config.RequestConfig
import org.apache.http.client.methods.AbstractExecutionAwareRequest
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.methods.HttpHead
import org.apache.http.client.methods.HttpPost
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.impl.client.HttpClientBuilder
import org.apache.http.impl.client.HttpClients
import org.apache.http.impl.client.LaxRedirectStrategy
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Service
import reactor.core.Disposable
import me.mnlr.vripper.event.Event
import kotlinx.coroutines.*
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.event.SettingsUpdateEvent
import org.apache.hc.client5.http.async.methods.SimpleRequestBuilder
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.classic.methods.HttpHead
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.classic.methods.HttpUriRequest
import org.apache.hc.client5.http.config.ConnectionConfig
import org.apache.hc.client5.http.config.RequestConfig
import org.apache.hc.client5.http.cookie.StandardCookieSpec
import org.apache.hc.client5.http.impl.DefaultRedirectStrategy
import org.apache.hc.client5.http.impl.async.HttpAsyncClientBuilder
import org.apache.hc.client5.http.impl.async.HttpAsyncClients
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManagerBuilder
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.pool.PoolConcurrencyPolicy
import org.apache.hc.core5.pool.PoolReusePolicy
import org.apache.hc.core5.util.TimeValue
import org.apache.hc.core5.util.Timeout
import java.net.URI
import java.util.concurrent.TimeUnit
@Service
class HTTPService(eventBus: EventBus, settingsService: SettingsService) {
private val disposable: Disposable
private lateinit var pcm: PoolingHttpClientConnectionManager
private lateinit var rc: RequestConfig
private var connectionTimeout: Int = settingsService.settings.connectionSettings.timeout
class HTTPService(
val eventBus: EventBus,
settingsService: SettingsService
) {
init {
disposable = eventBus
.flux()
.filter { it.kind == Event.Kind.SETTINGS_UPDATE }
.map { it.data as Settings }
.subscribe {
if (connectionTimeout != it.connectionSettings.timeout) {
connectionTimeout = it.connectionSettings.timeout
buildRequestConfig()
}
}
buildRequestConfig()
buildConnectionPool()
companion object {
const val USER_AGENT =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0"
}
@PreDestroy
private fun destroy() {
disposable.dispose()
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
private lateinit var pcm: PoolingAsyncClientConnectionManager
private lateinit var rc: RequestConfig
private lateinit var cc: ConnectionConfig
lateinit var clientBuilder: HttpAsyncClientBuilder
private var connectionTimeout: Long = settingsService.settings.connectionSettings.timeout
fun init() {
buildRequestConfig()
buildConnectionConfig()
buildConnectionPool()
buildClientBuilder()
coroutineScope.launch {
pcm.closeIdle(TimeValue.ofSeconds(60))
delay(15000)
}
coroutineScope.launch {
eventBus
.subscribe<SettingsUpdateEvent> {
if (connectionTimeout != it.settings.connectionSettings.timeout) {
connectionTimeout = it.settings.connectionSettings.timeout
buildRequestConfig()
buildConnectionConfig()
pcm.close()
buildConnectionPool()
buildClientBuilder()
}
}
}
}
private fun buildConnectionPool() {
pcm = PoolingHttpClientConnectionManager()
pcm.maxTotal = Int.MAX_VALUE
pcm.defaultMaxPerRoute = Int.MAX_VALUE
pcm = PoolingAsyncClientConnectionManagerBuilder.create()
.setPoolConcurrencyPolicy(PoolConcurrencyPolicy.STRICT)
.setConnPoolPolicy(PoolReusePolicy.LIFO)
.setDefaultConnectionConfig(cc)
.setMaxConnTotal(Int.MAX_VALUE)
.setMaxConnPerRoute(Int.MAX_VALUE)
.build()
}
private fun buildRequestConfig() {
rc = RequestConfig.custom()
.setConnectionRequestTimeout(connectionTimeout * 1000)
.setConnectTimeout(connectionTimeout * 1000)
.setSocketTimeout(connectionTimeout * 1000)
.setCookieSpec(CookieSpecs.STANDARD)
.setConnectionRequestTimeout(Timeout.ofSeconds(connectionTimeout))
.setCookieSpec(StandardCookieSpec.RELAXED)
.build()
}
@Scheduled(fixedDelay = 15000)
private fun idleConnectionMonitoring() {
pcm.closeIdleConnections(60, TimeUnit.SECONDS)
private fun buildConnectionConfig() {
cc = ConnectionConfig.custom()
.setConnectTimeout(Timeout.ofSeconds(connectionTimeout))
.setSocketTimeout(Timeout.ofSeconds(connectionTimeout))
.setTimeToLive(TimeValue.ofMinutes(10))
.build()
}
val client: HttpClientBuilder
get() = HttpClients.custom()
private fun buildClientBuilder() {
clientBuilder = HttpAsyncClients.custom()
.setConnectionManager(pcm)
.setRedirectStrategy(LaxRedirectStrategy())
.setRedirectStrategy(DefaultRedirectStrategy.INSTANCE)
.disableAutomaticRetries()
.setDefaultRequestConfig(rc)
}
fun buildHttpGet(url: String, context: HttpClientContext): HttpGet {
val httpGet = HttpGet(url.replace(" ", "+"))
@@ -83,6 +106,7 @@ class HTTPService(eventBus: EventBus, settingsService: SettingsService) {
}
fun buildHttpHead(url: String, context: HttpClientContext): HttpHead {
SimpleRequestBuilder.head(url)
val httpHead = HttpHead(url.replace(" ", "+"))
httpHead.addHeader("User-Agent", USER_AGENT)
addToContext(context, httpHead)
@@ -103,7 +127,7 @@ class HTTPService(eventBus: EventBus, settingsService: SettingsService) {
return httpGet
}
fun addToContext(context: HttpClientContext, request: AbstractExecutionAwareRequest) {
fun addToContext(context: HttpClientContext, request: HttpUriRequest) {
val contextAttributes =
context.getAttribute(
ContextAttributes.CONTEXT_ATTRIBUTES,
@@ -117,15 +141,10 @@ class HTTPService(eventBus: EventBus, settingsService: SettingsService) {
}
class ContextAttributes {
val requests: MutableList<AbstractExecutionAwareRequest> = mutableListOf()
val requests: MutableList<HttpUriRequest> = mutableListOf()
companion object {
const val CONTEXT_ATTRIBUTES = "CONTEXT_ATTRIBUTES"
}
}
companion object {
const val USER_AGENT =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
}
}
@@ -1,15 +1,13 @@
package me.mnlr.vripper.services
import me.mnlr.vripper.exception.HtmlProcessorException
import org.htmlcleaner.CleanerProperties
import org.htmlcleaner.DomSerializer
import org.htmlcleaner.HtmlCleaner
import org.springframework.stereotype.Service
import org.w3c.dom.Document
import me.mnlr.vripper.exception.HtmlProcessorException
import java.io.InputStream
@Service
class HtmlProcessorService {
object HtmlProcessorService {
@Throws(HtmlProcessorException::class)
fun clean(htmlContent: InputStream): Document {
return try {
@@ -1,38 +0,0 @@
package me.mnlr.vripper.services
import org.springframework.stereotype.Service
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.tasks.MetadataRunnable
import java.util.function.Consumer
@Service
class MetadataService(private val threadPoolService: ThreadPoolService) {
private val fetchingMetadata: MutableMap<String, MetadataRunnable> = mutableMapOf()
@Synchronized
fun startFetchingMetadata(postDownloadState: PostDownloadState) {
val runnable = MetadataRunnable(postDownloadState)
threadPoolService.generalExecutor.submit(runnable)
fetchingMetadata[postDownloadState.postId] = runnable
}
@Synchronized
fun stopFetchingMetadata(postIds: List<String>) {
val stopping: MutableList<MetadataRunnable> = ArrayList()
for ((key, value) in fetchingMetadata) {
if (postIds.contains(key)) {
stopping.add(value)
value.stop()
}
}
while (stopping.isNotEmpty()) {
stopping.removeIf { it.finished }
try {
Thread.sleep(100)
} catch (e: InterruptedException) {
Thread.currentThread().interrupt()
}
}
postIds.forEach(Consumer { key: String -> fetchingMetadata.remove(key) })
}
}
@@ -1,13 +1,10 @@
package me.mnlr.vripper.services
import org.springframework.stereotype.Service
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.model.Settings
import java.io.File
import java.nio.file.Path
@Service
class PathService {
object PathUtils {
private val log by LoggerDelegate()
fun calculateDownloadPath(forum: String, threadTitle: String, postTitle: String, postId: String, settings: Settings): Path {
@@ -22,18 +19,6 @@ class PathService {
return downloadDirectory
}
private fun makeDir(sourceFolder: File): File {
var counter = 1
var folder = sourceFolder
while (folder.exists()) {
folder = File(sourceFolder.toString() + '.' + counter++)
}
if (!folder.mkdirs()) {
throw RuntimeException(String.format("Failed to create the folder %s", sourceFolder))
}
return folder
}
/**
* Will sanitize the image name and remove extension
*
@@ -1,32 +1,32 @@
package me.mnlr.vripper.services
import jakarta.annotation.PreDestroy
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import net.jodah.failsafe.RetryPolicy
import net.jodah.failsafe.event.ExecutionAttemptedEvent
import org.springframework.stereotype.Service
import reactor.core.Disposable
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.model.Settings
import java.time.temporal.ChronoUnit
@Service
class RetryPolicyService(eventBus: EventBus, settingsService: SettingsService) {
class RetryPolicyService(
val eventBus: EventBus,
settingsService: SettingsService
) {
private val log by LoggerDelegate()
private val disposable: Disposable
private var maxAttempts: Int = settingsService.settings.connectionSettings.maxAttempts
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
init {
disposable = eventBus
.flux()
.filter { it.kind == Event.Kind.SETTINGS_UPDATE }
.map { it.data as Settings }
.subscribe {
if (maxAttempts != it.connectionSettings.maxAttempts) {
maxAttempts = it.connectionSettings.maxAttempts
fun init() {
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
if (maxAttempts != it.settings.connectionSettings.maxAttempts) {
maxAttempts = it.settings.connectionSettings.maxAttempts
}
}
}
}
fun <T> buildRetryPolicyForDownload(): RetryPolicy<T> {
@@ -46,9 +46,4 @@ class RetryPolicyService(eventBus: EventBus, settingsService: SettingsService) {
log.warn("#${e.attemptCount} tries failed", e.lastFailure)
}
}
@PreDestroy
private fun destroy() {
disposable.dispose()
}
}
@@ -5,32 +5,22 @@ import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.registerKotlinModule
import jakarta.annotation.PostConstruct
import jakarta.annotation.PreDestroy
import org.apache.commons.codec.digest.DigestUtils
import org.springframework.beans.factory.annotation.Value
import org.springframework.core.io.Resource
import org.springframework.stereotype.Service
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.exception.ValidationException
import me.mnlr.vripper.model.Settings
import org.apache.commons.codec.digest.DigestUtils
import java.io.FileWriter
import java.io.IOException
import java.nio.file.*
@Service
class SettingsService(
@Value("\${base.dir}") baseDir: String,
@Value("\${base.dir.name}") baseDirName: String,
private val eventBus: EventBus,
@Value("classpath:proxies.json") private val defaultProxies: Resource
) {
class SettingsService(private val eventBus: EventBus) {
private val log by LoggerDelegate()
private val configPath: Path
private val customProxiesPath: Path
private val configPath = Paths.get(baseDir, BASE_DIR_NAME, "config.yml")
private val customProxiesPath = Paths.get(baseDir, BASE_DIR_NAME, "proxies.json")
private val om = ObjectMapper(YAMLFactory())
private val proxies: MutableSet<String> = HashSet()
@@ -38,48 +28,51 @@ class SettingsService(
var settings = Settings()
init {
configPath = Paths.get(baseDir, baseDirName, "config.yml")
customProxiesPath = Paths.get(baseDir, baseDirName, "proxies.json")
om.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.findAndRegisterModules().registerKotlinModule()
init()
}
@PostConstruct
private fun init() {
loadViperProxies()
restore()
eventBus.publishEvent(Event(Event.Kind.SETTINGS_UPDATE, settings))
eventBus.publishEvent(SettingsUpdateEvent(settings))
}
private fun loadViperProxies() {
try {
defaultProxies.inputStream.use { defaultProxiesIs ->
val defaultProxies: List<String> = om.readValue(defaultProxiesIs)
SettingsService::class.java.getResourceAsStream("proxies.json")?.use {
val defaultProxies: List<String> = om.readValue(it)
val customProxies: List<String> = if (customProxiesPath.toFile()
.exists() && Files.isRegularFile(customProxiesPath)
) {
om.readValue(
customProxiesPath.toFile()
)
try {
om.readValue(
customProxiesPath.toFile()
)
} catch (e: Exception) {
emptyList()
}
} else {
try {
FileWriter(customProxiesPath.toFile()).use { fw ->
fw.append("[]").append(System.lineSeparator())
fw.flush()
}
} catch (e: Exception) {
log.warn(
"Unable to create " + customProxiesPath.toFile().absolutePath, e
)
}
emptyList()
}
try {
FileWriter(customProxiesPath.toFile()).use { fw ->
fw.append("[]").append(System.lineSeparator())
fw.flush()
}
} catch (e: IOException) {
log.warn(
"Unable to create " + customProxiesPath.toFile().absolutePath, e
)
}
proxies.addAll(customProxies)
proxies.addAll(defaultProxies)
proxies.addAll(customProxies)
}
} catch (e: IOException) {
} catch (e: Exception) {
log.error("Failed to load vipergirls proxies list", e)
}
proxies.add("https://vipergirls.to")
@@ -92,7 +85,8 @@ class SettingsService(
fun newSettings(settings: Settings) {
if (settings.viperSettings.login) {
if (this.settings.viperSettings.password != settings.viperSettings.password) {
settings.viperSettings.password = DigestUtils.md5Hex(settings.viperSettings.password)
settings.viperSettings.password =
DigestUtils.md5Hex(settings.viperSettings.password)
}
} else {
settings.viperSettings.username = ""
@@ -103,15 +97,17 @@ class SettingsService(
check(settings)
this.settings = settings
save()
eventBus.publishEvent(Event(Event.Kind.SETTINGS_UPDATE, settings))
eventBus.publishEvent(SettingsUpdateEvent(settings))
}
fun restore() {
private fun restore() {
try {
if (configPath.toFile().exists()) {
settings = om.readValue(configPath.toFile())
}
} catch (e: IOException) {
} catch (e: Exception) {
log.error("Failed restore user settings", e)
settings = Settings()
}
@@ -121,12 +117,8 @@ class SettingsService(
try {
check(settings)
} catch (e: ValidationException) {
log.error(
String.format(
"Your settings are invalid, either remove %s, or fix it", configPath
), e
)
SpringContext.close()
log.error("Your settings are invalid, using default settings", e)
settings = Settings()
}
save()
}
@@ -141,70 +133,64 @@ class SettingsService(
StandardOpenOption.TRUNCATE_EXISTING,
StandardOpenOption.SYNC
)
} catch (e: IOException) {
} catch (e: Exception) {
log.error("Failed to store user settings", e)
}
}
@PreDestroy
private fun destroy() {
save()
}
@Throws(ValidationException::class)
fun check(settings: Settings) {
val path: Path = try {
Paths.get(settings.downloadSettings.downloadPath)
} catch (e: InvalidPathException) {
throw ValidationException(String.format("%s is invalid", settings.downloadSettings.downloadPath))
throw ValidationException(
String.format(
"%s is invalid", settings.downloadSettings.downloadPath
)
)
}
if (!Files.exists(path)) {
throw ValidationException(String.format("%s does not exist", settings.downloadSettings.downloadPath))
throw ValidationException(
String.format(
"%s does not exist", settings.downloadSettings.downloadPath
)
)
} else if (!Files.isDirectory(path)) {
throw ValidationException(String.format("%s is not a directory", settings.downloadSettings.downloadPath))
throw ValidationException(
String.format(
"%s is not a directory", settings.downloadSettings.downloadPath
)
)
}
if(settings.downloadSettings.autoQueueThreshold < 0) {
if (settings.downloadSettings.autoQueueThreshold < 0) {
throw ValidationException("Invalid auto queue settings, value must be a positive integer")
}
if (settings.connectionSettings.maxTotalThreads < 0 || settings.connectionSettings.maxTotalThreads > 12) {
throw ValidationException(
String.format(
"Invalid max global concurrent download settings, values must be in [%d,%d]",
0,
12
)
"Invalid max global concurrent download settings, values must be in [0,12]"
)
}
if (settings.connectionSettings.maxThreads < 1 || settings.connectionSettings.maxThreads > 4) {
throw ValidationException(
String.format(
"Invalid max concurrent download settings, values must be in [%d,%d]", 1, 4
)
)
throw ValidationException("Invalid max concurrent download settings, values must be in [1,4]")
}
if (settings.connectionSettings.timeout < 1 || settings.connectionSettings.timeout > 300) {
throw ValidationException(
String.format(
"Invalid connection timeout settings, values must be in [%d,%d]", 1, 300
)
"Invalid connection timeout settings, values must be in [1,300]"
)
}
if (settings.connectionSettings.maxAttempts < 1 || settings.connectionSettings.maxAttempts > 10) {
throw ValidationException(
String.format(
"Invalid maximum attempts settings, values must be in [%d,%d]", 1, 10
)
)
throw ValidationException("Invalid maximum attempts settings, values must be in [1,10]")
}
if (settings.maxEventLog < 100 || settings.maxEventLog > 10000) {
throw ValidationException(
String.format(
"Invalid maximum event log record settings, values must be in [%d,%d]",
100,
10000
)
"Invalid maximum event log record settings, values must be in [100,10000]"
)
}
if (settings.clipboardSettings.pollingRate < 500) {
throw ValidationException(
"Invalid clipboard monitoring polling rate settings, values must be >= 500"
)
}
}
@@ -2,35 +2,34 @@ package me.mnlr.vripper.services
import com.github.benmanes.caffeine.cache.Caffeine
import com.github.benmanes.caffeine.cache.LoadingCache
import jakarta.annotation.PreDestroy
import org.springframework.stereotype.Service
import reactor.core.Disposable
import me.mnlr.vripper.event.Event
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.parser.ThreadLookupAPIParser
import java.util.*
import java.util.concurrent.ExecutionException
import java.util.concurrent.TimeUnit
@Service
class ThreadCacheService(eventBus: EventBus) {
class ThreadCacheService(val eventBus: EventBus) {
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
fun init() {
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
cache.invalidateAll()
}
}
}
private val cache: LoadingCache<String, ThreadItem> =
Caffeine.newBuilder().expireAfterWrite(5, TimeUnit.MINUTES)
.build { threadId: String ->
Caffeine.newBuilder().expireAfterWrite(5, TimeUnit.MINUTES).build { threadId: String ->
ThreadLookupAPIParser(threadId).parse()
}
private val disposable: Disposable
init {
disposable = eventBus.flux().filter { e: Event<*> -> e.kind == Event.Kind.SETTINGS_UPDATE }
.subscribe { cache.invalidateAll() }
}
@PreDestroy
private fun destroy() {
disposable.dispose()
}
@Throws(ExecutionException::class)
operator fun get(threadId: String): ThreadItem {
@@ -1,17 +0,0 @@
package me.mnlr.vripper.services
import org.springframework.stereotype.Service
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit
@Service
class ThreadPoolService {
val generalExecutor: ExecutorService = Executors.newFixedThreadPool(4)
fun destroy() {
generalExecutor.shutdown()
if (!generalExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
generalExecutor.shutdownNow()
}
}
}
@@ -1,77 +1,74 @@
package me.mnlr.vripper.services
import jakarta.annotation.PostConstruct
import jakarta.annotation.PreDestroy
import org.apache.http.NameValuePair
import org.apache.http.client.entity.UrlEncodedFormEntity
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.cookie.Cookie
import org.apache.http.impl.client.BasicCookieStore
import org.apache.http.message.BasicNameValuePair
import org.apache.http.util.EntityUtils
import org.springframework.stereotype.Service
import reactor.core.Disposable
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.event.VGUserLoginEvent
import me.mnlr.vripper.exception.VripperException
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.tasks.LeaveThanksRunnable
import org.apache.hc.client5.http.cookie.BasicCookieStore
import org.apache.hc.client5.http.cookie.Cookie
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import java.util.concurrent.CompletableFuture
@Service
class VGAuthService(
private val cm: HTTPService,
private val settingsService: SettingsService,
private val threadPoolService: ThreadPoolService,
private val eventBus: EventBus
) {
private val log by LoggerDelegate()
private val disposable: Disposable
final val context: HttpClientContext = HttpClientContext.create()
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
val context: HttpClientContext = HttpClientContext.create()
var loggedUser = ""
var authenticated = false
private var authenticated = false
init {
fun init() {
context.cookieStore = BasicCookieStore()
disposable = eventBus
.flux()
.filter { it.kind == Event.Kind.SETTINGS_UPDATE }
.subscribe { authenticate() }
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
authenticate(it.settings)
}
}
authenticate(settingsService.settings)
}
@PostConstruct
private fun init() {
authenticate()
}
@PreDestroy
private fun destroy() {
disposable.dispose()
}
fun authenticate() {
private fun authenticate(settings: Settings) {
authenticated = false
if (!settingsService.settings.viperSettings.login) {
if (!settings.viperSettings.login) {
log.debug("Authentication option is disabled")
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(Event(Event.Kind.VG_USER, loggedUser))
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
return
}
val username = settingsService.settings.viperSettings.username
val password = settingsService.settings.viperSettings.password
val username = settings.viperSettings.username
val password = settings.viperSettings.password
if (username.isEmpty() || password.isEmpty()) {
log.error("Cannot authenticate with ViperGirls credentials, username or password is empty")
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(Event(Event.Kind.VG_USER, loggedUser))
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
return
}
val postAuth =
cm.buildHttpPost(settingsService.settings.viperSettings.host + "/login.php?do=login", context)
cm.buildHttpPost(
settings.viperSettings.host + "/login.php?do=login",
context
)
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("vb_login_username", username))
params.add(BasicNameValuePair("cookieuser", "1"))
@@ -82,22 +79,28 @@ class VGAuthService(
} catch (e: Exception) {
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(Event(Event.Kind.VG_USER, loggedUser))
log.error("Failed to authenticate user with " + settingsService.settings.viperSettings.host, e)
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
log.error(
"Failed to authenticate user with " + settings.viperSettings.host,
e
)
return
}
postAuth.addHeader("Referer", settingsService.settings.viperSettings.host)
postAuth.addHeader("Referer", settings.viperSettings.host)
postAuth.addHeader(
"Host",
settingsService.settings.viperSettings.host.replace("https://", "").replace("http://", "")
settings.viperSettings.host.replace("https://", "")
.replace("http://", "")
)
val client = cm.client.build()
val client = cm.clientBuilder.build()
try {
client.execute(postAuth, context).use { response ->
if (response.statusLine.statusCode / 100 != 2) {
if (response.code / 100 != 2) {
throw VripperException(
String.format(
"Unexpected response code returned %s", response.statusLine.statusCode
"Unexpected response code returned %s", response.code
)
)
}
@@ -116,7 +119,7 @@ class VGAuthService(
log.error(
String.format(
"Failed to authenticate user with %s, missing vg_userid cookie",
settingsService.settings.viperSettings.host
settings.viperSettings.host
)
)
return
@@ -125,18 +128,24 @@ class VGAuthService(
} catch (e: Exception) {
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(Event(Event.Kind.VG_USER, loggedUser))
log.error("Failed to authenticate user with " + settingsService.settings.viperSettings.host, e)
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
log.error(
"Failed to authenticate user with " + settings.viperSettings.host,
e
)
return
}
authenticated = true
loggedUser = username
log.info(String.format("Authenticated: %s", username))
eventBus.publishEvent(Event(Event.Kind.VG_USER, loggedUser))
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
}
fun leaveThanks(post: PostDownloadState) {
threadPoolService.generalExecutor
.submit(LeaveThanksRunnable(post, authenticated, context))
fun leaveThanks(post: Post) {
CompletableFuture.runAsync(LeaveThanksRunnable(post, authenticated, context))
}
}
@@ -1,20 +1,23 @@
package me.mnlr.vripper.services
import org.springframework.stereotype.Service
import org.w3c.dom.Node
import org.w3c.dom.NodeList
import me.mnlr.vripper.exception.XpathException
import javax.xml.xpath.XPathConstants
import javax.xml.xpath.XPathFactory
@Service
class XpathService {
object XpathService {
private val xpath = XPathFactory.newInstance().newXPath()
@Throws(XpathException::class)
fun getAsNode(source: Node?, xpathExpression: String?): Node? {
return try {
xpath.compile(xpathExpression).evaluate(source, XPathConstants.NODE) as Node
val value = xpath.compile(xpathExpression).evaluate(source, XPathConstants.NODE)
if(value != null) {
value as Node
} else {
null
}
} catch (e: Exception) {
throw XpathException(e)
}
@@ -0,0 +1,15 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
object ImageTable : LongIdTable(name = "IMAGE", columnName = "ID") {
val current = long("CURRENT")
val host = varchar("HOST", 255)
val index = integer("INDEX")
val postId = varchar("POST_ID", 255)
val status = varchar("STATUS", 255)
val total = long("TOTAL")
val url = varchar("URL", 3000)
val thumbUrl = varchar("THUMB_URL", 3000)
val postIdRef = long("POST_ID_REF").references(PostTable.id, fkName = "IMAGE_POST_ID_REF_POST_ID_FK")
}
@@ -0,0 +1,22 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
import org.jetbrains.exposed.sql.javatime.datetime
import java.time.LocalDateTime
object PostTable : LongIdTable(name = "POST", columnName = "ID") {
val done = integer("DONE")
val hosts = varchar("HOSTS", 500)
val outputPath = varchar("OUTPUT_PATH", 500)
val postId = varchar("POST_ID", 255)
val status = varchar("STATUS", 255)
val threadId = varchar("THREAD_ID", 255)
val postTitle = varchar("POST_TITLE", 500)
val threadTitle = varchar("THREAD_TITLE", 500)
val forum = varchar("FORUM", 500)
val total = integer("TOTAL")
val url = varchar("URL", 3000)
val token = varchar("TOKEN", 500)
val addedAt = datetime("ADDED_AT").default(LocalDateTime.now())
val rank = integer("RANK").default(0)
}
@@ -0,0 +1,10 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
object ThreadTable : LongIdTable("THREAD", columnName = "ID") {
val total = integer("TOTAL").default(0)
val url = varchar("URL", 3000)
val threadId = varchar("THREAD_ID", 255)
val title = varchar("TITLE", 500)
}
@@ -1,38 +1,41 @@
package me.mnlr.vripper.tasks
import org.apache.http.NameValuePair
import org.apache.http.client.entity.UrlEncodedFormEntity
import org.apache.http.client.methods.HttpPost
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.impl.client.CloseableHttpClient
import org.apache.http.message.BasicNameValuePair
import org.apache.http.util.EntityUtils
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.HTTPService
import me.mnlr.vripper.services.SettingsService
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.io.UnsupportedEncodingException
class LeaveThanksRunnable(
private val postDownloadState: PostDownloadState,
private val post: Post,
private val authenticated: Boolean,
private val context: HttpClientContext
) : Runnable {
) : KoinComponent, Runnable {
private val log by LoggerDelegate()
private val cm: HTTPService = SpringContext.getBean(HTTPService::class.java)
private val settingsService: SettingsService = SpringContext.getBean(SettingsService::class.java)
private val eventRepository: LogEventRepository = SpringContext.getBean(LogEventRepository::class.java)
private val cm: HTTPService by inject()
private val settingsService: SettingsService by inject()
private val eventRepository: LogEventRepository by inject()
private val logEvent: LogEvent
init {
logEvent = eventRepository.save(
LogEvent(
type = LogEvent.Type.THANKS, status = PENDING, message = "Leaving thanks for $postDownloadState.url"
type = LogEvent.Type.THANKS,
status = PENDING,
message = "Leaving thanks for $post.url"
)
)
}
@@ -44,7 +47,7 @@ class LeaveThanksRunnable(
eventRepository.update(
logEvent.copy(
status = DONE,
message = "Will not send a like for ${postDownloadState.url}\nAuthentication with ViperGirls option is disabled"
message = "Will not send a like for ${post.url}\nAuthentication with ViperGirls option is disabled"
)
)
return
@@ -53,7 +56,7 @@ class LeaveThanksRunnable(
eventRepository.update(
logEvent.copy(
status = DONE,
message = "Will not send a like for ${postDownloadState.url}\nLeave thanks option is disabled"
message = "Will not send a like for ${post.url}\nLeave thanks option is disabled"
)
)
return
@@ -62,7 +65,7 @@ class LeaveThanksRunnable(
eventRepository.update(
logEvent.copy(
status = ERROR,
message = "Will not send a like for ${postDownloadState.url}\nYou are not authenticated"
message = "Will not send a like for ${post.url}\nYou are not authenticated"
)
)
return
@@ -73,12 +76,12 @@ class LeaveThanksRunnable(
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("do", "post_thanks_add"))
params.add(BasicNameValuePair("using_ajax", "1"))
params.add(BasicNameValuePair("p", postDownloadState.postId))
params.add(BasicNameValuePair("securitytoken", postDownloadState.token))
params.add(BasicNameValuePair("p", post.postId))
params.add(BasicNameValuePair("securitytoken", post.token))
try {
postThanks.entity = UrlEncodedFormEntity(params)
} catch (e: UnsupportedEncodingException) {
val error = "Request error for ${postDownloadState.url}"
val error = "Request error for ${post.url}"
log.error(error, e)
eventRepository.update(
logEvent.copy(
@@ -94,7 +97,7 @@ class LeaveThanksRunnable(
postThanks.addHeader(
"Host", settingsService.settings.viperSettings.host.replace("https://", "").replace("http://", "")
)
val client: CloseableHttpClient = cm.client.build()
val client: CloseableHttpClient = cm.clientBuilder.build()
client.execute(postThanks, context).use { response ->
try {
} finally {
@@ -103,7 +106,7 @@ class LeaveThanksRunnable(
}
eventRepository.update(logEvent.copy(status = DONE))
} catch (e: Exception) {
val error = "Failed to leave a thanks for $postDownloadState"
val error = "Failed to leave a thanks for $post"
log.error(error, e)
eventRepository.update(
logEvent.copy(
@@ -1,110 +0,0 @@
package me.mnlr.vripper.tasks
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.model.ThreadInput
import me.mnlr.vripper.repositories.impl.LogEventRepositoryImpl
import me.mnlr.vripper.services.SettingsService
import me.mnlr.vripper.services.ThreadPoolService
import java.time.LocalDateTime
import java.util.regex.Pattern
class LinkScanRunnable(private val urlList: List<String>) : Runnable {
private val log by LoggerDelegate()
private val settingsService: SettingsService =
SpringContext.getBean(SettingsService::class.java)
private val threadPoolService: ThreadPoolService =
SpringContext.getBean(ThreadPoolService::class.java)
private val eventRepository: LogEventRepositoryImpl =
SpringContext.getBean(LogEventRepositoryImpl::class.java)
private val logEvent: LogEvent
init {
logEvent = eventRepository.save(
LogEvent(
type = LogEvent.Type.SCAN,
status = PENDING,
time = LocalDateTime.now(),
message = "Links to scan:\n\t${urlList.joinToString("\n\t")}"
)
)
}
override fun run() {
try {
eventRepository.update(logEvent.copy(status = PROCESSING))
val threadInputList = mutableListOf<ThreadInput>()
val unsupported = mutableListOf<String>()
val unrecognized = mutableListOf<String>()
for (link in urlList) {
log.debug("Starting to process thread: $link")
if (!link.startsWith(settingsService.settings.viperSettings.host)) {
log.error("Unsupported link $link")
unsupported.add(link)
continue
}
var threadId: String
val m = Pattern.compile(
Pattern.quote(settingsService.settings.viperSettings.host) + "/threads/(\\d+).*"
).matcher(link)
if (m.find()) {
threadId = m.group(1)
} else {
log.error("Cannot retrieve thread id from URL $link")
unrecognized.add(link)
continue
}
threadInputList.add(ThreadInput(link = link, threadId = threadId))
}
val errorMessage: String = if (unsupported.isNotEmpty()) {
"""Unsupported links:
${unsupported.joinToString("\n\t")}
"""
} else if (unrecognized.isNotEmpty()) {
"""Unrecognized links:
${unrecognized.joinToString("\n\t")}
"""
} else {
""
}
if (errorMessage.isNotBlank()) {
eventRepository.update(
logEvent.copy(
status = ERROR, message = "Some links failed to be scanned: \n$errorMessage"
)
)
}
for (threadInput in threadInputList) {
// if (thread.postId.isNotBlank()) {
// threadPoolService.generalExecutor.submit(
// SinglePostRunnable(
// thread.threadId, thread.postId
// )
// )
// } else {
threadPoolService.generalExecutor.submit(
ThreadLookupRunnable(
threadInput,
settingsService.settings
)
)
// }
}
eventRepository.update(logEvent.copy(status = DONE))
} catch (e: Exception) {
val error = "Error when scanning links"
log.error(error, e)
eventRepository.update(
logEvent.copy(
status = ERROR, message = """
$error
${e.formatToString()}
""".trimIndent()
)
)
}
}
}
@@ -1,177 +0,0 @@
package me.mnlr.vripper.tasks
import org.apache.http.client.HttpClient
import org.apache.http.client.methods.CloseableHttpResponse
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.protocol.HttpClientContext
import org.apache.http.util.EntityUtils
import org.w3c.dom.Document
import org.w3c.dom.Node
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
import me.mnlr.vripper.entities.Metadata
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.*
import java.util.concurrent.atomic.AtomicBoolean
import java.util.stream.Collectors
class MetadataRunnable(private val postDownloadState: PostDownloadState) : Runnable {
private val log by LoggerDelegate()
private val dataTransaction: DataTransaction =
SpringContext.getBean(DataTransaction::class.java)
private val eventRepository: LogEventRepository =
SpringContext.getBean(LogEventRepository::class.java)
private val cm: HTTPService = SpringContext.getBean(HTTPService::class.java)
private val htmlProcessorService: HtmlProcessorService =
SpringContext.getBean(HtmlProcessorService::class.java)
private val xpathService: XpathService = SpringContext.getBean(XpathService::class.java)
private val metadataService: MetadataService =
SpringContext.getBean(MetadataService::class.java)
private val context: HttpClientContext = HttpClientContext.create()
private val logEvent: LogEvent
@Volatile
private var stopped = false
@Volatile
var finished = false
init {
val vgAuthService: VGAuthService = SpringContext.getBean(VGAuthService::class.java)
context.cookieStore = vgAuthService.context.cookieStore
context.setAttribute(
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES, HTTPService.ContextAttributes()
)
logEvent = eventRepository.save(
LogEvent(
type = LogEvent.Type.METADATA,
status = PENDING,
message = "Fetching metadata for " + postDownloadState.url
)
)
}
override fun run() {
try {
eventRepository.update(logEvent.copy(status = PROCESSING))
val metadata = fetchMetadata(postDownloadState.postId, postDownloadState.url)
dataTransaction.setMetadata(postDownloadState, metadata)
eventRepository.update(logEvent.copy(status = DONE))
} catch (e: Exception) {
val message = "Failed to fetch metadata for ${postDownloadState.url}"
log.error(message, e)
eventRepository.update(
logEvent.copy(
status = ERROR, message = """
$message
${e.formatToString()}
""".trimIndent()
)
)
} finally {
if (stopped) {
val message = "Fetching metadata for ${postDownloadState.url} interrupted"
eventRepository.update(logEvent.copy(status = DONE, message = message))
}
finished = true
metadataService.stopFetchingMetadata(listOf(postDownloadState.postId))
}
}
private fun fetchMetadata(postId: String, url: String): Metadata {
val httpGet: HttpGet = cm.buildHttpGet(url, context)
val connection: HttpClient = cm.client.build()
(connection.execute(httpGet, context) as CloseableHttpResponse).use { response ->
try {
if (response.statusLine.statusCode / 100 != 2) {
throw DownloadException("Unexpected response code '${response.statusLine.statusCode}' for $httpGet")
}
val document: Document =
htmlProcessorService.clean(response.entity.content)
val postNode: Node? =
xpathService.getAsNode(
document,
"//li[@id='post_$postId']/div[contains(@class, 'postdetails')]"
)
val postedBy: String = xpathService.getAsNode(
postNode,
"./div[contains(@class, 'userinfo')]//a[contains(@class, 'username')]//font"
)?.textContent?.trim() ?: ""
val metadata = Metadata()
metadata.postedBy = postedBy
val node: Node? = xpathService.getAsNode(
document, "//div[@id='post_message_$postId']"
)
if (node != null) {
metadata.resolvedNames = findTitleInContent(node)
} else {
metadata.resolvedNames = emptyList()
}
metadata.postId = postId
return metadata
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
}
private fun findTitleInContent(node: Node): List<String> {
val altTitle: MutableList<String> = ArrayList()
findTitle(node, altTitle, AtomicBoolean(true))
return altTitle.stream().distinct().collect(Collectors.toList())
}
private fun findTitle(node: Node, altTitle: MutableList<String>, keepGoing: AtomicBoolean) {
if (!keepGoing.get()) {
return
}
if (node.nodeName == "a" || node.nodeName == "img") {
keepGoing.set(false)
return
}
if (node.nodeType == Node.ELEMENT_NODE) {
for (i in 0 until node.childNodes.length) {
val item = node.childNodes.item(i)
findTitle(item, altTitle, keepGoing)
if (!keepGoing.get()) {
return
}
}
} else if (node.nodeType == Node.TEXT_NODE) {
val text = node.textContent.trim()
if (text.isNotBlank() && dictionary.stream().noneMatch { e: String ->
text.lowercase().contains(e.lowercase())
}) {
altTitle.add(text)
}
}
}
fun stop() {
stopped = true
val contextAttributes: HTTPService.ContextAttributes? = context.getAttribute(
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES,
HTTPService.ContextAttributes::class.java
)
if (contextAttributes != null) {
synchronized(contextAttributes.requests) {
for (request in contextAttributes.requests) {
request.abort()
}
}
}
}
companion object {
private val dictionary: List<String> =
mutableListOf("download", "link", "rapidgator", "filefactory", "filefox")
}
}
@@ -1,32 +1,32 @@
package me.mnlr.vripper.tasks
import me.mnlr.vripper.SpringContext
import me.mnlr.vripper.AppEndpointService
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.download.PostDownloadRunnable
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.model.ThreadInput
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.repositories.ThreadRepository
import me.mnlr.vripper.services.DataTransaction
import me.mnlr.vripper.services.SettingsService
import me.mnlr.vripper.services.ThreadCacheService
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.util.concurrent.CompletableFuture
class ThreadLookupRunnable(private val threadInput: ThreadInput, private val settings: Settings) : Runnable {
class ThreadLookupRunnable(private val threadId: String, private val settings: Settings) :
KoinComponent, Runnable {
private val log by LoggerDelegate()
private val dataTransaction = SpringContext.getBean(DataTransaction::class.java)
private val eventRepository = SpringContext.getBean(LogEventRepository::class.java)
private val threadCacheService = SpringContext.getBean(
ThreadCacheService::class.java
)
private val threadRepository = SpringContext.getBean(
ThreadRepository::class.java
)
private val downloadService = SpringContext.getBean(DownloadService::class.java)
private val dataTransaction by inject<DataTransaction>()
private val eventRepository by inject<LogEventRepository>()
private val settingsService by inject<SettingsService>()
private val threadCacheService by inject<ThreadCacheService>()
private val appEndpointService by inject<AppEndpointService>()
private val link: String =
"${settingsService.settings.viperSettings.host}/threads/$threadId"
private val logEvent: LogEvent
init {
@@ -34,7 +34,7 @@ class ThreadLookupRunnable(private val threadInput: ThreadInput, private val set
LogEvent(
type = LogEvent.Type.THREAD,
status = LogEvent.Status.PENDING,
message = "Processing multi-post link ${threadInput.link}"
message = "Processing multi-post link $link"
)
)
}
@@ -42,50 +42,38 @@ class ThreadLookupRunnable(private val threadInput: ThreadInput, private val set
override fun run() {
try {
eventRepository.update(logEvent.copy(status = PROCESSING))
val threadLookupResult = threadCacheService[threadInput.threadId]
if (threadLookupResult.postItemList.isEmpty()) {
val message = "Nothing found for " + threadInput.link
eventRepository.update(logEvent.copy(status = ERROR, message = message))
return
}
// if (threadItemList.size == 1) {
// threadPoolService.generalExecutor.submit(
// SinglePostRunnable(
// threadItemList[0].postId, threadItemList[0].threadId
// )
// )
// eventRepository.update(
// logEvent.copy(
// status = DONE,
// message = "Thread ${thread.link} is added to the queue"
// )
// )
// } else {
if (threadRepository.findByThreadId(threadInput.threadId).isEmpty) {
if (dataTransaction.findThreadByThreadId(threadId).isEmpty) {
val threadLookupResult = threadCacheService[threadId]
if (threadLookupResult.postItemList.isEmpty()) {
val message = "Nothing found for $link"
eventRepository.update(logEvent.copy(status = ERROR, message = message))
return
}
dataTransaction.save(
Thread(
link = threadInput.link,
threadId = threadInput.threadId,
title = threadLookupResult.title,
link = link,
threadId = threadId,
total = threadLookupResult.postItemList.size
)
)
eventRepository.update(
logEvent.copy(
status = DONE, message = "New thread ${threadInput.link} is added"
status = DONE, message = "New thread $link is added"
)
)
autostart(threadLookupResult)
} else {
log.info("Link ${threadInput.link} is already loaded")
log.info("Link $link is already loaded")
eventRepository.update(
logEvent.copy(
status = ERROR, message = "${threadInput.link} has already been added to the queue"
status = ERROR,
message = "$link has already been added to the queue"
)
)
}
// }
} catch (e: Exception) {
val error = "Error when adding multi-post link ${threadInput.link}"
val error = "Error when adding multi-post link $link"
log.error(error, e)
eventRepository.update(
logEvent.copy(
@@ -99,8 +87,13 @@ class ThreadLookupRunnable(private val threadInput: ThreadInput, private val set
}
private fun autostart(lookupResult: ThreadItem) {
if(lookupResult.postItemList.size <= settings.downloadSettings.autoQueueThreshold) {
downloadService.download(lookupResult.postItemList.map { Pair(it.threadId, it.postId) })
if (lookupResult.postItemList.size <= settings.downloadSettings.autoQueueThreshold) {
appEndpointService.threadRemove(listOf(lookupResult.threadId))
lookupResult.postItemList.forEach {
CompletableFuture.runAsync(PostDownloadRunnable(
it.threadId, it.postId
))
}
}
}
}
@@ -1,16 +0,0 @@
logging.level.org.springframework.web=INFO
logging.level.root=INFO
logging.level.org.springframework.web.socket.config.WebSocketMessageBrokerStats=ERROR
logging.file.name=${base.dir}/${base.dir.name}/vripper.log
server.port=${vripper.server.port:8080}
management.endpoints.web.exposure.include=shutdown
management.endpoint.shutdown.enabled=true
server.error.include-message=always
spring.liquibase.change-log=classpath:db/changelog/db.changelog-master.xml
spring.datasource.url=jdbc:hsqldb:file:${base.dir}/${base.dir.name}/db/vripper;hsqldb.lock_file=false
spring.datasource.username=SA
spring.datasource.password=lEtmEIn
spring.datasource.driver-class-name=org.hsqldb.jdbc.JDBCDriver
base.dir=${user.dir}
base.dir.name=${vripper.base.dir.name:vripper}
download.pool-size=12
@@ -4,7 +4,7 @@
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet author="vripper" id="0001">
<createTable tableName="POST">
<column name="ID" type="BIGINT">
<column autoIncrement="true" name="ID" type="BIGINT">
<constraints nullable="false" primaryKey="true" primaryKeyName="POST_PK"/>
</column>
<column name="DONE" type="INT">
@@ -13,7 +13,7 @@
<column name="HOSTS" type="VARCHAR(500)">
<constraints nullable="false"/>
</column>
<column name="POST_FOLDER_NAME" type="VARCHAR(500)"/>
<column name="OUTPUT_PATH" type="VARCHAR(500)"/>
<column name="POST_ID" type="VARCHAR(255)">
<constraints nullable="false"/>
</column>
@@ -41,7 +41,7 @@
<column name="TOKEN" type="VARCHAR(500)">
<constraints nullable="false"/>
</column>
<column defaultValueComputed="CURRENT_TIMESTAMP" name="ADDED_ON" type="TIMESTAMP">
<column defaultValueComputed="CURRENT_TIMESTAMP" name="ADDED_AT" type="TIMESTAMP">
<constraints nullable="false"/>
</column>
<column defaultValue="0" name="RANK" type="INT">
@@ -50,7 +50,8 @@
</createTable>
<createTable tableName="METADATA">
<column name="POST_ID_REF" type="BIGINT">
<constraints nullable="false" primaryKey="true" primaryKeyName="METADATA_PK" references="POST(ID)" foreignKeyName="METADATA_POST_ID_REF_POST_ID_FK"/>
<constraints foreignKeyName="METADATA_POST_ID_REF_POST_ID_FK" nullable="false" primaryKey="true"
primaryKeyName="METADATA_PK" references="POST(ID)"/>
</column>
<column name="POST_ID" type="VARCHAR(255)">
<constraints nullable="false"/>
@@ -59,7 +60,7 @@
<column name="RESOLVED_NAMES" type="VARCHAR(16777216)"/>
</createTable>
<createTable tableName="IMAGE">
<column name="ID" type="BIGINT">
<column autoIncrement="true" name="ID" type="BIGINT">
<constraints nullable="false" primaryKey="true" primaryKeyName="IMAGE_PK"/>
</column>
<column name="CURRENT" type="BIGINT">
@@ -83,23 +84,30 @@
<column name="URL" type="VARCHAR(3000)">
<constraints nullable="false"/>
</column>
<column name="THUMB_URL" type="VARCHAR(3000)">
<constraints nullable="false"/>
</column>
<column name="POST_ID_REF" type="BIGINT">
<constraints nullable="false" references="POST(ID)" foreignKeyName="IMAGE_POST_ID_REF_POST_ID_FK"/>
<constraints foreignKeyName="IMAGE_POST_ID_REF_POST_ID_FK" nullable="false"
references="POST(ID)"/>
</column>
</createTable>
<createTable tableName="THREAD">
<column name="ID" type="BIGINT">
<column autoIncrement="true" name="ID" type="BIGINT">
<constraints nullable="false" primaryKey="true" primaryKeyName="THREAD_PK"/>
</column>
<column name="TOTAL" type="INT">
<constraints nullable="false"/>
</column>
<column name="LINK" type="VARCHAR(3000)">
<column name="URL" type="VARCHAR(3000)">
<constraints nullable="false"/>
</column>
<column name="THREAD_ID" type="VARCHAR(255)">
<constraints nullable="false"/>
</column>
<column name="TITLE" type="VARCHAR(500)">
<constraints nullable="false"/>
</column>
</createTable>
<createTable tableName="EVENT">
<column name="ID" type="BIGINT">
@@ -119,12 +127,6 @@
</column>
</createTable>
</changeSet>
<changeSet author="vripper" id="0002">
<createSequence incrementBy="1" sequenceName="SEQ_IMAGE" startValue="1"/>
<createSequence incrementBy="1" sequenceName="SEQ_POST" startValue="1"/>
<createSequence incrementBy="1" sequenceName="SEQ_THREAD" startValue="1"/>
<createSequence incrementBy="1" sequenceName="SEQ_EVENT" startValue="1"/>
</changeSet>
<changeSet author="vripper" id="0003">
<createIndex indexName="IMAGE_POST_ID_IDX" tableName="IMAGE">
<column name="POST_ID"/>
@@ -1,24 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource=
"org/springframework/boot/logging/logback/defaults.xml"/>
<include resource=
"org/springframework/boot/logging/logback/file-appender.xml"/>
<include resource=
"org/springframework/boot/logging/logback/console-appender.xml"/>
<appender name="ASYNC-FILE" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE"/>
</appender>
<appender name="ASYNC-CONSOLE" class="ch.qos.logback.classic.AsyncAppender" >
<appender-ref ref="CONSOLE"/>
</appender>
<root level="INFO">
<appender-ref ref="ASYNC-FILE"/>
<appender-ref ref="ASYNC-CONSOLE"/>
</root>
</configuration>
@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<!-- <logger level="debug" name="org.apache.http.headers"/>-->
<root level="INFO">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
+31 -48
View File
@@ -4,29 +4,31 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
<relativePath/>
<version>3.0.6</version> <!-- lookup parent from repository -->
<artifactId>vripper-project</artifactId>
<groupId>me.mnlr</groupId>
<version>4.4.0</version>
</parent>
<groupId>me.mnlr.vripper</groupId>
<artifactId>vripper-gui</artifactId>
<version>4.0.0</version>
<name>vripper-gui</name>
<description>vripper-gui</description>
<properties>
<javafx.version>21</javafx.version>
</properties>
<dependencies>
<dependency>
<artifactId>spring-boot-starter</artifactId>
<groupId>org.springframework.boot</groupId>
</dependency>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<artifactId>kotlin-stdlib</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core-jvm</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-javafx</artifactId>
<version>1.7.3</version>
</dependency>
<dependency>
<artifactId>javafx-base</artifactId>
<groupId>org.openjfx</groupId>
@@ -99,56 +101,37 @@
<dependency>
<artifactId>vripper-core</artifactId>
<groupId>me.mnlr.vripper</groupId>
<version>${project.version}</version>
</dependency>
<dependency>
<artifactId>spring-boot-starter-test</artifactId>
<groupId>org.springframework.boot</groupId>
<scope>test</scope>
<version>4.4.0</version>
</dependency>
</dependencies>
<properties>
<java.version>17</java.version>
<kotlin.version>1.8.21</kotlin.version>
<javafx.version>20.0.1</javafx.version>
<maven.deploy.skip>false</maven.deploy.skip>
</properties>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<sourceDirs>
<source>src/main/kotlin</source>
<source>target/generated-sources/annotations</source>
</sourceDirs>
</configuration>
</execution>
</executions>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<artifactId>kotlin-maven-allopen</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
<distributionManagement>
<repository>
<id>github</id>
<name>GitHub death-claw Apache Maven Packages</name>
<url>https://maven.pkg.github.com/death-claw/vripper-project</url>
</repository>
</distributionManagement>
</project>
@@ -1,76 +0,0 @@
package me.mnlr.vripper
import javafx.application.Application
import javafx.scene.image.Image
import javafx.stage.Stage
import javafx.stage.WindowEvent
import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.context.ConfigurableApplicationContext
import me.mnlr.vripper.event.ApplicationInitialized
import me.mnlr.vripper.gui.Styles
import me.mnlr.vripper.view.LoadingView
import tornadofx.*
import kotlin.reflect.KClass
import kotlin.system.exitProcess
@SpringBootApplication
class VripperGuiApplication : App(
LoadingView::class, Styles::class
) { //The application class must be a TornadoFX application and it must have the main view
private lateinit var context: ConfigurableApplicationContext //We are going to set application context here
private var initialized = false
override fun start(stage: Stage) {
with(stage) {
width = 1200.0
height = 600.0
icons.addAll(
listOf(
Image("icons/16x16.png"),
Image("icons/32x32.png"),
Image("icons/48x48.png"),
Image("icons/64x64.png"),
Image("icons/128x128.png"),
Image("icons/256x256.png"),
Image("icons/512x512.png"),
Image("icons/1024x1024.png")
)
)
}
stage.addEventFilter(WindowEvent.WINDOW_SHOWN) {
val contextInitThread = Thread {
context =
SpringApplication.run(this.javaClass) //We start the application context and let Spring Boot to initilaize itself
context.autowireCapableBeanFactory.autowireBean(this) //We ask the context to inject all needed dependencies into the current instence (if needed)
FX.dicontainer = object :
DIContainer { // Here we have to implement an interface for TornadoFX DI support
override fun <T : Any> getInstance(type: KClass<T>): T =
context.getBean(type.java) // We find dependencies directly in Spring's application context
override fun <T : Any> getInstance(type: KClass<T>, name: String): T =
context.getBean(name, type.java)
}
fire(ApplicationInitialized)
initialized = true
}
contextInitThread.apply {
this.name = "Spring init Thread"
}.start()
}
super.start(stage)
}
override fun stop() { // On stop, we have to stop spring as well
super.stop()
if (initialized) {
context.close()
}
exitProcess(0)
}
}
fun main(args: Array<String>) {
Application.launch(me.mnlr.vripper.VripperGuiApplication::class.java, *args)
}
@@ -1,62 +0,0 @@
package me.mnlr.vripper.controller
import me.mnlr.vripper.event.Event
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.formatSI
import me.mnlr.vripper.model.DownloadSpeed
import me.mnlr.vripper.model.GlobalState
import me.mnlr.vripper.model.GlobalStateModel
import me.mnlr.vripper.services.GlobalStateService
import tornadofx.*
class GlobalStateController : Controller() {
private val globalStateService: GlobalStateService by di()
private val eventBus: EventBus by di()
var globalState: GlobalStateModel = GlobalStateModel(0, 0, 0, "", "")
init {
val currentState = globalStateService.get()
globalState.apply {
running = currentState.running
remaining = currentState.remaining
error = currentState.error
loggedUser = currentState.loggedUser
downloadSpeed = currentState.downloadSpeed
}
eventBus.flux().subscribe {
if (it!!.kind == Event.Kind.DOWNLOAD_STATUS
) {
val newState = it.data as GlobalState
globalState.apply {
running = newState.running
remaining = newState.remaining
error = newState.error
}
}
}
eventBus.flux().subscribe {
if (it!!.kind == Event.Kind.VG_USER
) {
val user = it.data as String
globalState.apply {
loggedUser = user
}
}
}
eventBus.flux().subscribe {
if (it!!.kind == Event.Kind.BYTES_PER_SECOND
) {
val speed =
DownloadSpeed((it.data as Long).formatSI())
globalState.apply {
downloadSpeed = speed.speed
}
}
}
}
}
@@ -1,31 +0,0 @@
package me.mnlr.vripper.controller
import me.mnlr.vripper.entities.ImageDownloadState
import me.mnlr.vripper.model.ImageModel
import me.mnlr.vripper.repositories.ImageRepository
import tornadofx.Controller
import java.util.*
class ImageController : Controller() {
private val imageRepository: ImageRepository by di()
fun findImages(postId: String): List<ImageModel> {
return imageRepository.findByPostId(postId).map(::mapper)
}
fun findImageById(id: Long): Optional<ImageModel> {
return imageRepository.findById(id).map(::mapper)
}
private fun mapper(it: ImageDownloadState): ImageModel {
return ImageModel(
it.id!!,
it.index + 1,
it.url,
if (it.current == 0L && it.total == 0L) 0.0 else (it.current.toDouble() / it.total),
it.status.stringValue,
it.postId
)
}
}
@@ -1,69 +0,0 @@
package me.mnlr.vripper.controller
import me.mnlr.vripper.AppEndpointService
import me.mnlr.vripper.entities.PostDownloadState
import me.mnlr.vripper.model.PostModel
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import tornadofx.Controller
import java.time.format.DateTimeFormatter
import java.util.*
class PostController : Controller() {
private val postDownloadStateRepository: PostDownloadStateRepository by di()
private val appEndpointService: AppEndpointService by di()
private val dateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd hh:mm:ss")
fun scan(postLinks: String) {
appEndpointService.scanLinks(postLinks)
}
fun start(postIdList: List<String>) {
appEndpointService.restartAll(postIdList)
}
fun startAll() {
appEndpointService.restartAll()
}
fun delete(postIdList: List<String>) {
appEndpointService.remove(postIdList)
}
fun stop(postIdList: List<String>) {
appEndpointService.stopAll(postIdList)
}
fun clearPosts(): List<String> {
return appEndpointService.clearCompleted()
}
fun stopAll() {
appEndpointService.stopAll(null)
}
fun findAllPosts(): List<PostModel> {
return postDownloadStateRepository.findAll().map(::mapper)
}
fun findById(id: Long): Optional<PostModel> {
return postDownloadStateRepository.findById(id).map(::mapper)
}
private fun mapper(it: PostDownloadState): PostModel {
return PostModel(
it.postId,
it.postTitle,
if (it.done == 0 && it.total == 0) 0.0 else (it.done.toDouble() / it.total),
it.status.stringValue,
it.url,
it.done,
it.total,
it.hosts.joinToString(separator = "||"),
it.addedOn.format(dateTimeFormatter),
it.rank + 1,
it.downloadDirectory
)
}
}
@@ -1,69 +0,0 @@
package me.mnlr.vripper.controller
import javafx.scene.control.Alert
import me.mnlr.vripper.exception.ValidationException
import me.mnlr.vripper.model.ConnectionSettings
import me.mnlr.vripper.model.DownloadSettings
import me.mnlr.vripper.model.settings.DownloadSettingsModel
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.model.ViperSettings
import me.mnlr.vripper.model.settings.ConnectionSettingsModel
import me.mnlr.vripper.model.settings.ViperSettingsModel
import me.mnlr.vripper.services.SettingsService
import tornadofx.Controller
import tornadofx.alert
import tornadofx.warning
class SettingsController : Controller() {
private val settingsService: SettingsService by di()
fun findDownloadSettings(): DownloadSettings {
return settingsService.settings.downloadSettings
}
fun findConnectionSettings(): ConnectionSettings {
return settingsService.settings.connectionSettings
}
fun findViperGirlsSettings(): ViperSettings {
return settingsService.settings.viperSettings
}
fun saveNewSettings(
downloadSettingsModel: DownloadSettingsModel,
connectionSettingsModel: ConnectionSettingsModel,
viperSettingsModel: ViperSettingsModel
) {
settingsService.newSettings(Settings().apply {
downloadSettings = DownloadSettings(
downloadSettingsModel.downloadPath,
downloadSettingsModel.autoStart,
downloadSettingsModel.autoQueueThreshold,
downloadSettingsModel.forceOrder,
downloadSettingsModel.forumSubfolder,
downloadSettingsModel.threadSubLocation,
downloadSettingsModel.clearCompleted,
downloadSettingsModel.appendPostId
)
connectionSettings = ConnectionSettings(
connectionSettingsModel.maxThreads,
connectionSettingsModel.maxTotalThreads,
connectionSettingsModel.timeout,
connectionSettingsModel.maxAttempts,
)
viperSettings = ViperSettings(
viperSettingsModel.login,
viperSettingsModel.username,
viperSettingsModel.password,
viperSettingsModel.thanks,
viperSettingsModel.host,
)
})
}
fun getProxies(): List<String> {
return settingsService.getProxies()
}
}

Some files were not shown because too many files have changed in this diff Show More