Compare commits

...
70 Commits
Author SHA1 Message Date
death-claw 745bc17606 v5 wip 2023-10-13 00:00:01 +01:00
death-claw 04e37c83d7 v5 wip 2023-10-12 01:15:13 +01:00
death-claw 9bb865d98e v5 wip 2023-10-11 21:16:57 +01:00
death-claw 1625c70ce0 v5 wip 2023-10-11 00:50:03 +01:00
death-claw 63bc3ed3cc v5 wip 2023-10-10 19:17:16 +01:00
death-claw b1565e3a71 v5 wip 2023-10-09 08:46:06 +01:00
death-claw b1ef2243a0 v5 wip 2023-10-08 23:54:47 +01:00
death-claw 1d1c8918f0 v5 wip 2023-10-08 20:32:09 +01:00
death-claw cc0972adbc v5 wip 2023-10-08 09:30:06 +01:00
death-claw d9e8e08adb v5 wip 2023-10-06 09:31:02 +01:00
death-claw c08c430160 v5 wip 2023-10-04 00:39:09 +01:00
death-claw 4572ca61a3 v5 wip 2023-10-03 00:58:20 +01:00
death-claw ca23c80c24 v5 wip 2023-10-02 00:33:06 +01:00
l1xanderandGitHub f0c2de829a Fix for images on Pimpandhost throwing exception (#127)
Attempts to fix the following exception:


Failed to download http://pimpandhost.com/image/39446205-medium.html
me.mnlr.vripper.exception. DownloadException: me.mnlr.vripper.exception.HostException: Xpath //img[contains(@class, 'original')]' cannot be found in 'http://pimpandhost.com/image/39446205-medium.html' at me.mnlr.vripper.download.ImageDownloadRunnable.download(Image Download Runnable.kt:88)
at me.mnlr.vripper.download.ImageDownloadRunnable.run(ImageDownloadRunnable.kt:151)
at net.jodah.failsafe.Functions.lambda$toSupplier$10(Functions.java:262)
at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48)
at net.jodah.failsafe.Retry Policy Executor.lambda$supply$0(RetryPolicyExecutor.java:66)
at net.jodah.failsafe.Execution.executeSync(Execution.java:128)
at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:379)
at net.jodah.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:212)
at me.mnlr.vripper.download.DownloadService.schedule$lambda$28 (DownloadService.kt:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: me.mnlr.vripper.exception.HostException: Xpath //img[contains(@class, 'original')]' cannot be found in 'http://pimpandhost.com/image/39446205-medium.html'
at me.mnlr.vripper.host.PimpandhostHost.resolve(PimpandhostHost.kt:46)
at me.mnlr.vripper.host.Host.download Internal(Host.kt:62)
at me.mnlr.vripper.download.ImageDownloadRunnable.download (ImageDownloadRunnable.kt:72)
11 more
2023-09-22 10:04:19 +01:00
death-claw 08fcd7a87d version bump 2023-09-14 22:17:52 +01:00
death-clawandGitHub f39c7a15c7 fixes #123 (#126) 2023-09-14 21:53:49 +01:00
death-clawandGitHub 1c239d435b fixes #122 (#125) 2023-09-14 21:53:35 +01:00
death-clawandGitHub d8cba97d05 fixes #120 (#124) 2023-09-14 21:52:08 +01:00
death-clawandGitHub a362a3fa7f fixes #118 (#119) 2023-09-03 14:04:50 +01:00
death-clawandGitHub 4163f2a53f fixes #116 (#117) 2023-09-03 14:02:44 +01:00
death-claw e556dd1804 fix packaging 2023-09-01 15:24:32 +01:00
death-claw 96ad55b930 fix packaging 2023-09-01 15:17:33 +01:00
death-claw 9dc1a10255 fix packaging 2023-09-01 15:02:03 +01:00
death-claw db4a041bc4 fix packaging 2023-09-01 14:48:17 +01:00
death-claw 661f591bf5 fix packaging 2023-09-01 14:35:44 +01:00
death-claw e81bcfbfee fix packaging 2023-09-01 14:27:27 +01:00
death-claw e8ba07aebd version bump 2023-09-01 13:57:02 +01:00
death-clawandGitHub 9930226dc0 Enhance packaging (#114) 2023-09-01 13:54:51 +01:00
death-clawandGitHub 48491767cb remove threads when auto queue activates (#112) 2023-09-01 13:05:16 +01:00
death-clawandGitHub 49229b9dac open gallery folder option missing on right click (#111) 2023-09-01 01:01:27 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ecf3287e61 Bump socket.io-parser from 4.2.2 to 4.2.4 in /vripper-web-ui (#108)
Bumps [socket.io-parser](https://github.com/socketio/socket.io-parser) from 4.2.2 to 4.2.4.
- [Release notes](https://github.com/socketio/socket.io-parser/releases)
- [Changelog](https://github.com/socketio/socket.io-parser/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io-parser/compare/4.2.2...4.2.4)

---
updated-dependencies:
- dependency-name: socket.io-parser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-01 00:10:35 +01:00
death-clawandGitHub 206b305ae6 have an option to save temp files on different drive (#110) 2023-09-01 00:09:49 +01:00
death-clawandGitHub 70e7a87189 dependencies update (#109) 2023-08-31 23:28:59 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5aef1c0a89 Bump word-wrap from 1.2.3 to 1.2.4 in /vripper-web-ui (#106)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 22:43:30 +01:00
death-claw 7745abbb8a Update package.yml and version bump 2023-05-26 00:22:12 +01:00
death-clawandGitHub e577dba7c9 #100 wip (#101) 2023-05-24 22:20:02 +01:00
death-clawandGitHub ed63d5c029 Feature/issue 97/enhance lock file mechanism (#99)
* fixes #97

* fixes #97
2023-05-24 20:42:25 +01:00
death-clawandGitHub 2aa82a73a0 fixes #97 (#98) 2023-05-24 20:27:31 +01:00
death-clawandGitHub cbe5800c08 Update and rename maven-publish.yml to release.yml 2023-05-21 17:28:38 +01:00
death-clawandGitHub b200226060 Update maven-publish.yml 2023-05-21 16:56:53 +01:00
death-clawandGitHub 0d22b054c9 v4.1.0 (#96) 2023-05-21 16:42:11 +01:00
death-clawandGitHub 9d18dc8897 Update maven-publish.yml 2023-05-21 16:38:06 +01:00
death-claw f42d412b86 fixes #89 (#95) 2023-05-21 15:55:42 +01:00
death-clawandGitHub f8370801c2 fixes #89 (#94) 2023-05-21 13:18:23 +01:00
death-clawandGitHub 51dcaa1946 fixes #88 (#93) 2023-05-14 21:44:44 +01:00
death-clawandGitHub a833a0b1b2 Update maven-publish.yml 2023-05-13 17:20:40 +01:00
death-clawandGitHub 79739b80f9 fixes #90 (#92) 2023-05-13 17:16:00 +01:00
death-claw 26cf60b741 version bump 2023-05-06 21:26:39 +01:00
death-claw bc45deed2a resolves #86 2023-05-06 21:22:18 +01:00
death-claw b1e3353a1d v4.0.0 2023-05-06 20:03:12 +01:00
death-claw c018033d2f Code enhancements 2021-05-31 20:22:16 +01:00
death-claw b0f5a15943 Add support for dpic.me 2021-05-31 19:06:41 +01:00
death-claw 8db4d406ad v3.5.4 2021-05-29 10:32:26 +01:00
death-claw 47f1705abb Fix download queue bug 2021-05-29 10:30:18 +01:00
death-claw 816fb84549 v3.5.3 2021-05-24 15:17:30 +01:00
death-claw 5213ed7129 Fix icon 2021-05-24 15:15:49 +01:00
death-claw 2483c5fa00 v3.5.2 2021-05-24 14:59:32 +01:00
death-claw dfc5810fd3 Add missing icon 2021-05-24 14:57:47 +01:00
death-claw fad418134e Update donation infos 2021-05-24 14:56:00 +01:00
death-claw 9dd91bfc07 v3.5.1 2021-05-22 15:51:05 +01:00
death-claw 5bd42e2ce5 Merge remote-tracking branch 'origin/master' 2021-05-22 15:44:22 +01:00
death-claw e18832b154 v3.5.0 2021-05-21 20:11:16 +01:00
death-claw 41feb53a83 Bug fix 2021-05-20 23:46:59 +01:00
death-claw 6f166b42c4 Enhance metadata interruption
Add Added on and Order columns
2021-05-20 21:25:52 +01:00
death-claw e8ee579a37 Save window size and position
Resize confirmation box
2021-05-18 21:10:42 +01:00
death-claw 41ce99995b Download service enhancements 2021-05-10 21:19:40 +01:00
death-claw 5caeef0e70 v3.4.2 2021-05-10 11:18:43 +01:00
death-claw 96ef0cb234 Fix race condition 2021-05-10 11:17:02 +01:00
death-claw 641b3e3e7b v3.4.1 2021-05-10 00:21:23 +01:00
death-claw 08de4dc5b0 Fix icons 2021-05-10 00:19:42 +01:00
559 changed files with 28881 additions and 25470 deletions
+55
View File
@@ -0,0 +1,55 @@
name: Package
on:
workflow_dispatch:
jobs:
build:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build with Maven
run: mvn -B install --file pom.xml
- name: Rename GUI artifact
run: mv vripper-gui/target/vripper-gui-*.jar vripper-gui/target/vripper-gui.jar
- name: Rename WEB artifact
run: mv vripper-web/target/vripper-web-*.jar vripper-web/target/vripper-web.jar
- if: matrix.os == 'ubuntu-latest'
name: Package Linux binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage --type deb "@jpackage.cfg" "@jpackage-linux.cfg"
jpackage --type rpm "@jpackage.cfg" "@jpackage-linux.cfg"
ls -la dist
- if: matrix.os == 'windows-latest'
name: Package Windows binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage "@jpackage.cfg" "@jpackage-windows.cfg"
dir dist
- if: matrix.os == 'macos-latest'
name: Package macOS binaries
run: |
cd jpackage
cp ../vripper-gui/target/vripper-gui.jar jar/vripper-gui.jar
jpackage "@jpackage.cfg" "@jpackage-macos.cfg"
ls -la dist
+144
View File
@@ -0,0 +1,144 @@
name: Release
on:
release:
types: [created]
jobs:
build:
strategy:
matrix:
os: [ ubuntu-latest, windows-latest, macos-latest ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build Core Jar
run: mvn -B -q install --file vripper-core/pom.xml
- name: Build GUI Jar
run: mvn -B -q install --file vripper-gui/pom.xml
- name: Rename GUI Jar
run: mv vripper-gui/target/vripper-gui-${{ github.event.release.tag_name }}.jar vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar
- if: matrix.os == 'ubuntu-latest'
name: Upload GUI Jar
uses: softprops/action-gh-release@v1
with:
files: vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar
# Start building WEB jar in ubuntu only
- if: matrix.os == 'ubuntu-latest'
name: Build WEB Jar
run: |
mvn -B -q install --file vripper-web-ui/pom.xml
mvn -B -q install --file vripper-web/pom.xml
- if: matrix.os == 'ubuntu-latest'
name: Rename WEB Jar
run: mv vripper-web/target/vripper-web-${{ github.event.release.tag_name }}.jar vripper-web/target/vripper-noarch-web-${{ github.event.release.tag_name }}.jar
- if: matrix.os == 'ubuntu-latest'
name: Upload WEB Jar
uses: softprops/action-gh-release@v1
with:
files: vripper-web/target/vripper-noarch-web-${{ github.event.release.tag_name }}.jar
# End building WEB jar in ubuntu only
- name: Prepare Packaging
run: |
cp vripper-gui/target/vripper-noarch-gui-${{ github.event.release.tag_name }}.jar jpackage/jar/vripper-gui.jar
- if: matrix.os == 'ubuntu-latest'
name: Package Linux
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.png
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-linux.cfg" --type deb
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-linux.cfg" --type rpm
mv dist/vripper-${{ github.event.release.tag_name }}-1.x86_64.rpm dist/vripper-linux-${{ github.event.release.tag_name }}.x86_64.rpm
mv dist/vripper_${{ github.event.release.tag_name }}-1_amd64.deb dist/vripper-linux-${{ github.event.release.tag_name }}_amd64.deb
- if: matrix.os == 'windows-latest'
name: Package Windows
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.ico
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-windows.cfg" --type msi
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-windows.cfg" --type exe
cd dist
ren VRipper-${{ github.event.release.tag_name }}.msi vripper-windows-installer-${{ github.event.release.tag_name }}.msi
ren VRipper-${{ github.event.release.tag_name }}.exe vripper-windows-installer-${{ github.event.release.tag_name }}.exe
- if: matrix.os == 'macos-latest'
name: Package macOS
run: |
cd jpackage
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-app-image.cfg" --icon icon.icns
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-macos.cfg" --type pkg
jpackage --app-version ${{ github.event.release.tag_name }} "@jpackage.cfg" "@jpackage-macos.cfg" --type dmg
mv dist/VRipper-${{ github.event.release.tag_name }}.pkg dist/vripper-macos-${{ github.event.release.tag_name }}.pkg
mv dist/VRipper-${{ github.event.release.tag_name }}.dmg dist/vripper-macos-${{ github.event.release.tag_name }}.dmg
- if: matrix.os == 'ubuntu-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper'
filename: 'vripper-linux-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'windows-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper'
filename: 'vripper-windows-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'macos-latest'
name: Zip portable
uses: thedoctor0/zip-release@0.7.1
with:
type: 'zip'
directory: 'jpackage/dist'
path: 'VRipper.app'
filename: 'vripper-macos-portable-${{ github.event.release.tag_name }}.zip'
- if: matrix.os == 'ubuntu-latest'
name: Upload DEB and RPM package for Linux
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-linux-${{ github.event.release.tag_name }}.x86_64.rpm
jpackage/dist/vripper-linux-${{ github.event.release.tag_name }}_amd64.deb
jpackage/dist/vripper-linux-portable-${{ github.event.release.tag_name }}.zip
- if: matrix.os == 'windows-latest'
name: Upload packages for Windows
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-windows-installer-${{ github.event.release.tag_name }}.msi
jpackage/dist/vripper-windows-installer-${{ github.event.release.tag_name }}.exe
jpackage/dist/vripper-windows-portable-${{ github.event.release.tag_name }}.zip
- if: matrix.os == 'macos-latest'
name: Upload package for macOS
uses: softprops/action-gh-release@v1
with:
files: |
jpackage/dist/vripper-macos-${{ github.event.release.tag_name }}.pkg
jpackage/dist/vripper-macos-${{ github.event.release.tag_name }}.dmg
jpackage/dist/vripper-macos-portable-${{ github.event.release.tag_name }}.zip
-724
View File
@@ -1,724 +0,0 @@
# Changelog
## [3.4.0] - 2021-05-09
### Changed
- Fix potential issue with imagebam host
- Enhance download queue logic
- Code source enhancements
## [3.3.8] - 2021-05-05
### Changed
- Fix imagebam host
- Fix font path
## [3.3.6] - 2021-03-20
### Changed
- Optimize build size
## [3.3.5] - 2021-03-18
### Changed
- Bug fixe
## [3.3.4] - 2021-03-18
### Changed
- Bug fixes
## [3.3.3] - 2021-03-13
### Changed
- Remove unnecessary event logs
## [3.3.2] - 2021-03-11
### Changed
- Fix a bug with event logs
- Change event names and add some icons
## [3.3.1] - 2021-02-18
### Added
- Add about menu
### Changed
- Fix README.md
## [3.3.0] - 2021-02-16
### Changed
- Add support for proxies
- Add a Event Logs
- Fix bugs with the database
## [3.2.3] - 2021-01-13
### Changed
- Finished posts are not cleared from the UI
## [3.2.2] - 2021-01-13
### Changed
- Set total download to max value
## [3.2.1] - 2021-01-12
### Added
- Add settings for connection timeout
- Add settings for maximum attempts
### Changed
- Enhance posts rename logic
- Dependencies update
- Switch reactive from rx-java to reactor
## [3.2.0] - 2020-11-16
### Changed
- Fix build issue
## [3.1.0] - 2020-11-16
### Changed
- UI revamp
- Fix bugs
## [3.0.11] - 2020-10-31
### Added
- Support for vipr.im
## [3.0.10] - 2020-10-04
### Changed
- Fix bug on settings update
## [3.0.9] - 2020-09-27
### Changed
- Revert to appending post id to download folders
- Fix a bug that lead to adding duplicate posts
- Remove old unused settings
## [3.0.8] - 2020-08-22
### Changed
- Bug fix which breaks concurrent downloads
## [3.0.7] - 2020-08-22
### Changed
- Fix MacOs build
## [3.0.6] - 2020-08-18
### Changed
- Fix TurboImageHost
## [3.0.5] - 2020-08-17
### Changed
- Update packages
- Host class refactoring
## [3.0.4] - 2020-08-16
### Changed
- Improve rename UI
## [3.0.3] - 2020-08-16
### Changed
- Fix bugs with electron app
## [3.0.2] - 2020-08-15
### Changed
- Fix bugs with download queue
## [3.0.1] - 2020-08-03
### Changed
- Fix bugs with download queue
## [3.0.0] - 2020-08-03
### Changed
- Major rewrites
- Introducing a proper database for persistence
## [2.12.1] - 2020-05-22
### Changed
- Fix alternative name bug
## [2.12.0] - 2020-05-21
### Changed
- Use new user hash from the api
## [2.11.7] - 2020-05-08
### Changed
- Fix bug with paths creation
- Move Open Link button
- Rename rename all title
## [2.11.6] - 2020-05-04
### Changed
- PNG filenames are now correctly named
- Fix issue with special characters in post titles
## [2.11.5] - 2020-05-03
### Changed
- Reduce rows buffer size
## [2.11.4] - 2020-05-03
### Changed
- Code cleanup
- Dependencies update
## [2.11.3] - 2020-04-30
### Changed
- Source code restructure
- UI fixes and enhancements
### Added
- Add menu entry to rename a gallery to first alternative title
- Add menu on toolbar for massive rename to first alternative title
## [2.11.2] - 2020-04-26
### Changed
- Bug fixes
## [2.11.1] - 2020-04-25
### Changed
- Code source maintenance
### Added
- Added option to rename gallery
## [2.11.0] - 2020-03-29
### Changed
- Upgrade Electron to v8
- Upgrade Angular to v9
- App settings are now portable in a json file
- General code source maintenance
### Added
- Added Support for pixroute.com
- Added grab post title from thread option
## [2.10.8] - 2020-01-25
### Changed
- Fix app url
## [2.10.7] - 2020-01-25
### Changed
- Drop SockJs and switch to WebSocket
- Autoreconnect front to back
## [2.10.6] - 2020-01-19
### Changed
- Limit cache size
## [2.10.5] - 2020-01-18
### Changed
- Bug fixes
## [2.10.4] - 2020-01-18
### Changed
- Zero value for total max download will disable the limit
- Add system notification
- Bug fixes
## [2.10.3] - 2020-01-16
### Changed
- Some bug fixes
- Enhance download queue logic
- Clear cache button
- Fix the partial status
- Sanitize URLs with spaces
### Added
- Support for Postimg host
- Support for Imagevenue host
## [2.10.2] - 2019-12-29
### Changed
- Upgrade electron to v7
- Bug fix
## [2.10.1] - 2019-12-27
### Changed
- Bug fix
## [2.10.0] - 2019-12-27
### Changed
- Fix spring dependency bug
### Added
- Add support for imgspice
## [2.9.0] - 2019-12-27
### Changed
- Fix build issue
## [2.9.0] - 2019-12-27
### Changed
- Decrease timeout for fast fail
### Added
- Allow many hosts to run at the same time
- Add title to gallery
- Add global concurrent downloads
- Add counter badge in link collector
## [2.8.1] - 2019-12-24
### Changed
- Fix issue with path creation
## [2.8.0] - 2019-12-24
### Added
- Photo gallery feature
## [2.7.1] - 2019-12-21
### Added
- Add version to title bar
## [2.7.0] - 2019-12-21
### Changed
- Add missing icon
### Added
- Add support for pixxxels
## [2.6.0] - 2019-12-21
### Changed
- Disable download button when nothing selected in Link collector grab menu
- Better quality app icon
- Implement context menu on download tab
- Add stop remove and start button to context menu
- Link will not be removed automatically from link collector tab
- Some UI fixes
## [2.5.1] - 2019-12-09
### Changed
- Disable download button when nothing selected in Link collector grab menu
### Added
- Add hosts column in Link collector grab menu
## [2.5.0] - 2019-12-08
### Changed
- Change loading screen
- UI Fixes
- Fix bug with previews
### Added
- Add link collector feature
- Add support for multiple links at a time
## [2.4.1] - 2019-11-26
### Changed
- Increase attempts for establishing ws connection between front and back end
- Change default download folder to Home folder (installer), App folder (portable)
## [2.4.0] - 2019-11-23
### Changed
- Pimpandhost Host support
## [2.3.2] - 2019-11-23
### Changed
- Fix a bug with the scan component
## [2.3.1] - 2019-11-22
### Changed
- Fix connexion reset issue
- Fix Post Status when autostart option is selected
- Add stop all button
## [2.3.0] - 2019-11-21
### Changed
- UI enhancements and overhaul
- Stability fixes with HTTP connections
- MacOS official support
- Use local sockjs and material icons font
## [2.2.5] - 2019-11-09
### Changed
- Exclude thumbs download for imagezilla and pixhost
- Change retry on fail logic (may help avoid having download errors)
## [2.2.4] - 2019-10-05
### Added
- AppImage target for linux
## [2.2.3] - 2019-10-01
### Changed
- Strip libjvm.so for debian
## [2.2.2] - 2019-10-01
### Changed
- Update product name
## [2.2.1] - 2019-10-01
### Changed
- Update dependency list for deb package
## [2.2.0] - 2019-10-01
### Added
- Embedd JRE within the app
### Changed
- Fix 'Start' on pending items
- Upgrade to Java 11
## [2.1.0] - 2019-09-30
### Added
- Add a create a subfolder per thread option
## [2.0.4] - 2019-09-27
### Changed
- Fix some issues with ImxHost
## [2.0.3] - 2019-09-23
### Changed
- Update deb dependencies
## [2.0.2] - 2019-09-23
### Changed
- Add portable mode
## [2.0.1] - 2019-09-22
### Changed
- Change app name
## [2.0.0] - 2019-09-22
### Added
- Add sub folder option
- Add clear finished option
### Changed
- UI enhancements
- Bug fixes
- Change data and log paths
## [1.6.5] - 2019-09-18
### Changed
- electron update
## [1.6.4] - 2019-09-18
### Changed
- Change menu item name
## [1.6.3] - 2019-09-18
### Changed
- Better multithread management
- Fix partial state UI
## [1.6.2] - 2019-09-17
### Changed
- Rework UI colors
- Fix scan input bug
## [1.6.1] - 2019-09-16
### Changed
- Rework UI
- Fix clipboard monitoring
## [1.6.0] - 2019-09-15
### Changed
- Rework the UI
- Add a dark theme
- Fix the force ordering bug
- Other bug fixes
## [1.5.2] - 2019-09-01
### Added
- imx.to host fix
## [1.5.1] - 2019-08-24
### Added
- Bug fixes
## [1.5.0] - 2019-08-24
### Added
- Display logged in user
- Add post details when clicking on main list items (Status, Post URL, Host)
- Add open download location on main list items menu
- Add support for multi-post threads (Show a menu to select posts to download)
- Add link to image from details list
## [1.4.10] - 2019-08-15
### Changed
- Fix a rename issue on windows
## [1.4.9] - 2019-08-13
### Changed
- Update Built-By to be generic
## [1.4.8] - 2019-08-13
### Changed
- Add option to force image ordering
## [1.4.7] - 2019-08-13
### Changed
- Fixed a bug in image filename extensions
## [1.4.6] - 2019-08-06
### Changed
- Fixed a bug when creating image file names
- Remove unnecessary logs
- Fix data.json init
## [1.4.5] - 2019-08-03
### Changed
- Add min width and height for electron app
### Added
- Add download speed to status bar
## [1.4.4] - 2019-08-01
### Changed
- Fix names for ImgboxHost
- Enhance UI
- Minor fixes
### Added
- Add start and stop button
## [1.4.3] - 2019-07-14
### Changed
- Update some messages
- Fix scrolling issue
## [1.4.2] - 2019-07-14
### Changed
- Fix layout issue in web mode
## [1.4.1] - 2019-07-14
### Changed
- Change title's font
### Added
- Add borders
## [1.4.0] - 2019-07-14
### Changed
- Better error handling on startup
- Better support for linux AppImage
### Added
- Clear all completed
- Add clipboard support
- Add option to remove all posts
- Add confirmation on remove
## [1.3.0-rc0] - 2019-07-04
### Changed
- Change default setting for autostart false -> true
- Change default setting for max concurrent downloads 1 -> 4
### Added
- Desktop app built with electron
- Add the option to remove a post
## [1.2.0] - 2019-06-23
### Added
- Add a shutdown button to gracefully shutdown the app
- Open and redirect the browser to the app address on startup
- Add the option to run the app in headless mode (-Djava.awt.headless=true)
## [1.1.1] - 2019-06-22
### Changed
- Fix a stability issue with progress updates when the app is runninh for a long time
## [1.0.0] - 2019-06-22
### Added
- Add real time communication for progress updates via websocket (SockJS)
- Add settings for download path, max concurrent download, enable/disable integration with vipergirls
- Add integration with vipergitls.to (possibility to auto leave thanks after grab post)
- Add support for acidimg.cc, imagezilla.net, imgbox.com, imx.to, pixhost.to, turboimagehost.com
+8 -19
View File
@@ -1,30 +1,19 @@
# VRipper!
This is my spin for a cross platform gallery ripper app for [vipergirls](https://vipergirls.to) website.
The purpose of this project is to build a robust and clean application to conveniently download photo galleries using modern web technologies, Java + Spring boot for the back end and angular + electron for the front end.
This is my spin for a cross-platform gallery ripper for [vipergirls.to](https://vipergirls.to).
## How to build
You need a recent version of maven 3.6.1+.
The application support 2 build modes:
- Desktop app: Self contained app, built with electron includes Java runtimes.
- Server app: Depends on Java runtimes, you need a web browser to access the app UI.
You need JDK 17 and a recent version of maven 3.6.1+
Most people will be interested only on the desktop app, however if you have a nas or a server, the server app is there for you.
To build run the following:
To build the server app:
mvn clean install
mvn clean install -DskipTests
Build artifact is located under
To build the desktop app:
vripper-project\vripper-gui\target\vripper-gui-{{version}}.jar
mvn clean install -Pelectron -DskipTests
Copy the artifact into any other folder and run:
Maven will automatically handle front end compilation. However, for development, you will need to install a recent
version of nodejs on your system.
If you feel generous and want to support me, you can donate some coins, and I would be very grateful 🙏
I accept Bitcoin (BTC) for donations, following is my address
bc1q9jzzw3st49r9vmxgtylq4vrd67cl7zfclf3lsq
java -jar vripper-gui-{{version}}.jar

Before

Width:  |  Height:  |  Size: 303 KiB

After

Width:  |  Height:  |  Size: 303 KiB

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

+2
View File
@@ -0,0 +1,2 @@
--type app-image
--java-options "-Dbase.dir=${user.dir}"
+7
View File
@@ -0,0 +1,7 @@
--icon icon.png
--java-options "-Dbase.dir=${user.home}/.config"
--linux-package-name vripper
--linux-app-release 1
--linux-menu-group Utility
--linux-app-category java
--linux-shortcut
+4
View File
@@ -0,0 +1,4 @@
--icon icon.icns
--java-options "-Dbase.dir=${user.home}/.config"
--mac-package-identifier me.mnlr.vripper.vripper-gui
--mac-package-name VRipper
+8
View File
@@ -0,0 +1,8 @@
--icon icon.ico
--java-options "-Dbase.dir=${user.home}"
--win-dir-chooser
--win-menu
--win-per-user-install
--win-shortcut
--win-shortcut-prompt
--win-update-url https://github.com/death-claw/vripper-project/releases
+8
View File
@@ -0,0 +1,8 @@
--input jar
--main-jar vripper-gui.jar
--add-modules java.base,java.desktop,java.sql,jdk.unsupported,jdk.crypto.ec,java.security.jgss
--jlink-options "--strip-native-commands --strip-debug --no-man-pages --no-header-files --compress=2"
--name VRipper
--description "Image ripper tool for vipergirls"
--dest dist
--vendor "death-claw"
+141 -18
View File
@@ -1,20 +1,143 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>tn.mnlr</groupId>
<artifactId>vripper</artifactId>
<version>3.4.0</version>
<packaging>pom</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.5</version>
<relativePath/>
</parent>
<modules>
<module>vripper-server</module>
<module>vripper-ui</module>
<module>vripper-electron</module>
</modules>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>me.mnlr</groupId>
<artifactId>vripper-project</artifactId>
<modules>
<module>vripper-core</module>
<!-- <module>vripper-web-ui</module>-->
<!-- <module>vripper-web</module>-->
<module>vripper-gui</module>
</modules>
<packaging>pom</packaging>
<version>4.4.0</version>
<properties>
<java.version>17</java.version>
<maven.deploy.skip>false</maven.deploy.skip>
<timestamp>${maven.build.timestamp}</timestamp>
<maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss</maven.build.timestamp.format>
<maven.compiler.target>17</maven.compiler.target>
<maven.compiler.source>17</maven.compiler.source>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
<version>1.4.11</version>
</dependency>
<dependency>
<artifactId>kotlin-stdlib</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>1.9.10</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core-jvm</artifactId>
<version>1.7.3</version>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-jdbc</artifactId>
<version>0.41.1</version>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-java-time</artifactId>
<version>0.41.1</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>3.5.10</version>
</dependency>
<dependency>
<groupId>io.insert-koin</groupId>
<artifactId>koin-core-jvm</artifactId>
<version>3.4.2</version>
</dependency>
<dependency>
<artifactId>jackson-module-kotlin</artifactId>
<groupId>com.fasterxml.jackson.module</groupId>
<version>2.15.2</version>
</dependency>
<dependency>
<artifactId>h2</artifactId>
<groupId>com.h2database</groupId>
<version>2.2.224</version>
</dependency>
<dependency>
<artifactId>httpclient</artifactId>
<groupId>org.apache.httpcomponents</groupId>
<version>4.5.14</version>
</dependency>
<dependency>
<artifactId>htmlcleaner</artifactId>
<groupId>net.sourceforge.htmlcleaner</groupId>
<version>2.26</version>
</dependency>
<dependency>
<artifactId>failsafe</artifactId>
<groupId>net.jodah</groupId>
<version>2.4.4</version>
</dependency>
<dependency>
<artifactId>caffeine</artifactId>
<groupId>com.github.ben-manes.caffeine</groupId>
<version>3.1.8</version>
</dependency>
<dependency>
<artifactId>jackson-databind</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>5.13.0</version>
</dependency>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
<version>1.9.10</version>
</dependency>
<dependency>
<artifactId>jcl-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
<version>1.7.32</version>
</dependency>
<dependency>
<artifactId>httpclient5</artifactId>
<groupId>org.apache.httpcomponents.client5</groupId>
<version>5.2.1</version>
</dependency>
<dependency>
<artifactId>commons-codec</artifactId>
<groupId>commons-codec</groupId>
<version>1.16.0</version>
</dependency>
<dependency>
<artifactId>liquibase-core</artifactId>
<groupId>org.liquibase</groupId>
<version>4.24.0</version>
</dependency>
</dependencies>
</dependencyManagement>
</project>
View File
View File
+124
View File
@@ -0,0 +1,124 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>me.mnlr</groupId>
<artifactId>vripper-project</artifactId>
<version>4.4.0</version>
</parent>
<groupId>me.mnlr.vripper</groupId>
<artifactId>vripper-core</artifactId>
<name>vripper-core</name>
<description>vripper-core</description>
<dependencies>
<dependency>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
</dependency>
<dependency>
<artifactId>kotlin-stdlib</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core-jvm</artifactId>
</dependency>
<dependency>
<artifactId>kotlin-reflect</artifactId>
<groupId>org.jetbrains.kotlin</groupId>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.exposed</groupId>
<artifactId>exposed-java-time</artifactId>
</dependency>
<dependency>
<groupId>io.insert-koin</groupId>
<artifactId>koin-core-jvm</artifactId>
</dependency>
<dependency>
<artifactId>jackson-module-kotlin</artifactId>
<groupId>com.fasterxml.jackson.module</groupId>
</dependency>
<dependency>
<artifactId>h2</artifactId>
<groupId>com.h2database</groupId>
</dependency>
<dependency>
<artifactId>httpclient5</artifactId>
<groupId>org.apache.httpcomponents.client5</groupId>
</dependency>
<dependency>
<artifactId>jcl-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
</dependency>
<dependency>
<artifactId>htmlcleaner</artifactId>
<groupId>net.sourceforge.htmlcleaner</groupId>
</dependency>
<dependency>
<artifactId>failsafe</artifactId>
<groupId>net.jodah</groupId>
</dependency>
<dependency>
<artifactId>caffeine</artifactId>
<groupId>com.github.ben-manes.caffeine</groupId>
</dependency>
<dependency>
<artifactId>jackson-databind</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</dependency>
<dependency>
<artifactId>commons-codec</artifactId>
<groupId>commons-codec</groupId>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
</dependency>
<dependency>
<artifactId>liquibase-core</artifactId>
<groupId>org.liquibase</groupId>
</dependency>
</dependencies>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<sourceDirs>
<source>src/main/kotlin</source>
<source>target/generated-sources/annotations</source>
</sourceDirs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
@@ -0,0 +1,139 @@
package me.mnlr.vripper
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.download.PostDownloadRunnable
import me.mnlr.vripper.exception.PostParseException
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.*
import me.mnlr.vripper.tasks.ThreadLookupRunnable
import java.util.*
import java.util.concurrent.CompletableFuture
import java.util.regex.Pattern
class AppEndpointService(
private val downloadService: DownloadService,
private val dataTransaction: DataTransaction,
private val threadCacheService: ThreadCacheService,
private val eventRepository: LogEventRepository,
private val settingsService: SettingsService
) {
private val log by LoggerDelegate()
@Synchronized
fun scanLinks(postLinks: String) {
if (postLinks.isBlank()) {
log.warn("Nothing to scan")
return
}
val urlList = postLinks.split(Pattern.compile("\\r?\\n")).dropLastWhile { it.isEmpty() }
.map { it.trim() }.filter { it.isNotEmpty() }
for (link in urlList) {
log.debug("Starting to process thread: $link")
if (!link.startsWith(settingsService.settings.viperSettings.host)) {
continue
}
var threadId: String
var postId: String?
val m = Pattern.compile(
Pattern.quote(settingsService.settings.viperSettings.host) + "/threads/(\\d+)((.*p=)(\\d+))?"
).matcher(link)
if (m.find()) {
threadId = m.group(1)
postId = m.group(4)
if (postId == null) {
CompletableFuture.runAsync(ThreadLookupRunnable(
threadId, settingsService.settings
))
} else {
CompletableFuture.runAsync(PostDownloadRunnable(
threadId, postId
))
}
} else {
log.error("Cannot retrieve thread id from URL $link")
continue
}
}
}
@Synchronized
fun restartAll(posIds: List<String> = listOf()) {
downloadService.restartAll(posIds)
}
@Synchronized
fun download(posts: List<Pair<String, String>>) {
posts.forEach {
CompletableFuture.runAsync(PostDownloadRunnable(
it.first, it.second
))
}
}
@Synchronized
fun stopAll(postIdList: List<String>?) {
downloadService.stopAll(postIdList)
}
@Synchronized
fun remove(postIdList: List<String>) {
downloadService.stopAll(postIdList)
dataTransaction.removeAll(postIdList)
}
@Synchronized
fun clearCompleted(): List<String> {
return dataTransaction.clearCompleted()
}
@Synchronized
@Throws(PostParseException::class)
fun grab(threadId: String): List<PostItem> {
return try {
val thread = dataTransaction.findThreadByThreadId(threadId).orElseThrow {
PostParseException(
String.format(
"Unable to find links for threadId = %s", threadId
)
)
}
val threadLookupResult = threadCacheService[thread.threadId]
threadLookupResult.postItemList.ifEmpty {
log.error(
String.format(
"Failed to get links for threadId = %s", threadId
)
)
throw PostParseException(
String.format(
"Failed to get links for threadId = %s", threadId
)
)
}
} catch (e: Exception) {
throw PostParseException(
String.format(
"Failed to get links for threadId = %s, %s", threadId, e.message
)
)
}
}
@Synchronized
fun threadRemove(threadIdList: List<String>) {
threadIdList.forEach {
dataTransaction.removeThread(it)
}
}
@Synchronized
fun threadClear() {
dataTransaction.clearQueueLinks()
}
fun logClear() {
eventRepository.deleteAll()
}
}
@@ -0,0 +1,13 @@
package me.mnlr.vripper
import java.nio.file.Files
import java.nio.file.Path
object ApplicationProperties {
const val BASE_DIR_NAME: String = "vripper"
val baseDir: String = System.getProperty("base.dir", System.getProperty("user.dir"))
init {
Files.createDirectories(Path.of(baseDir, BASE_DIR_NAME))
}
}
@@ -0,0 +1,106 @@
package me.mnlr.vripper
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.host.*
import me.mnlr.vripper.repositories.*
import me.mnlr.vripper.repositories.impl.*
import me.mnlr.vripper.services.*
import org.koin.dsl.bind
import org.koin.dsl.module
val coreModule = module {
single<EventBus> {
EventBus
}
single<SettingsService> {
SettingsService(get())
}
single<ImageRepository> {
ImageRepositoryImpl()
}
single<PostDownloadStateRepository> {
PostDownloadStateRepositoryImpl()
}
single<MetadataRepository> {
MetadataRepositoryImpl()
}
single<ThreadRepository> {
ThreadRepositoryImpl()
}
single<LogEventRepository> {
LogEventRepositoryImpl()
}
single<DataTransaction> {
DataTransaction(get(), get(), get(), get(), get(), get())
}
single<RetryPolicyService> {
RetryPolicyService(get(), get())
}
single<HTTPService> {
HTTPService(get(), get())
}
single<VGAuthService> {
VGAuthService(get(), get(), get())
}
single<ThreadCacheService> {
ThreadCacheService(get())
}
single<DownloadService> {
DownloadService(get(), get(), get(), get(), get())
}
single<DownloadSpeedService> {
DownloadSpeedService(get())
}
single<AppEndpointService> {
AppEndpointService(get(), get(), get(), get(), get())
}
single {
AcidimgHost(get(), get(), get())
} bind Host::class
single {
DPicMeHost(get(), get(), get())
} bind Host::class
single {
ImageBamHost(get(), get(), get())
} bind Host::class
single {
ImageTwistHost(get(), get(), get())
} bind Host::class
single {
ImageVenueHost(get(), get(), get())
} bind Host::class
single {
ImageZillaHost(get(), get(), get())
} bind Host::class
single {
ImgboxHost(get(), get(), get())
} bind Host::class
single {
ImgSpiceHost(get(), get(), get())
} bind Host::class
single {
ImxHost(get(), get(), get())
} bind Host::class
single {
PimpandhostHost(get(), get(), get())
} bind Host::class
single {
PixhostHost(get(), get(), get())
} bind Host::class
single {
PixRouteHost(get(), get(), get())
} bind Host::class
single {
PixxxelsHost(get(), get(), get())
} bind Host::class
single {
PostImgHost(get(), get(), get())
} bind Host::class
single {
TurboImageHost(get(), get(), get())
} bind Host::class
single {
ViprImHost(get(), get(), get())
} bind Host::class
}
@@ -0,0 +1,47 @@
package me.mnlr.vripper
import java.io.PrintWriter
import java.io.StringWriter
import java.security.MessageDigest
fun Throwable.formatToString(): String {
return StringWriter().use { stringWriter ->
PrintWriter(stringWriter).use { printWriter ->
this.printStackTrace(printWriter)
printWriter.flush()
stringWriter.flush()
stringWriter.toString()
}
}
}
fun Long.formatSI(): String {
return humanReadableByteCount(this, false)
}
private fun humanReadableByteCount(bytes: Long, si: Boolean): String {
val unit = if (si) 1000 else 1024
if (bytes < unit) return "$bytes B"
val exp = (Math.log(bytes.toDouble()) / Math.log(unit.toDouble())).toInt()
val pre = (if (si) "kMGTPE" else "KMGTPE")[exp - 1].toString() + if (si) "" else "i"
return String.format("%.1f %sB", bytes / Math.pow(unit.toDouble(), exp.toDouble()), pre)
}
fun getExtension(fileName: String): String {
return if (fileName.contains(".")) fileName.substring(fileName.lastIndexOf(".") + 1) else ""
}
fun getFileNameWithoutExtension(fileName: String): String {
return if (fileName.contains(".")) fileName.substring(
0,
fileName.lastIndexOf(".")
) else fileName
}
fun String.hash256(): String {
val bytes = this.toByteArray()
val md = MessageDigest.getInstance("SHA-256")
val digest = md.digest(bytes)
return digest.fold("") { str, it -> str + "%02x".format(it) }
}
@@ -0,0 +1,16 @@
package me.mnlr.vripper.delegate
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import kotlin.properties.ReadOnlyProperty
import kotlin.reflect.KProperty
class LoggerDelegate : ReadOnlyProperty<Any?, Logger> {
private var logger: Logger? = null
override operator fun getValue(thisRef: Any?, property: KProperty<*>): Logger {
if (logger == null) {
logger = LoggerFactory.getLogger(thisRef!!.javaClass)
}
return logger!!
}
}
@@ -0,0 +1,273 @@
package me.mnlr.vripper.download
import kotlinx.coroutines.*
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.ErrorCountEvent
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.QueueStateEvent
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.model.ErrorCount
import me.mnlr.vripper.model.QueueState
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.DataTransaction
import me.mnlr.vripper.services.RetryPolicyService
import me.mnlr.vripper.services.SettingsService
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.RetryPolicy
import java.util.concurrent.locks.ReentrantLock
import java.util.stream.Collectors
import kotlin.concurrent.withLock
class DownloadService(
private val settingsService: SettingsService,
private val dataTransaction: DataTransaction,
private val retryPolicyService: RetryPolicyService,
private val eventRepository: LogEventRepository,
private val eventBus: EventBus
) {
private val maxPoolSize: Int = 12
private val log by LoggerDelegate()
// Class fields
private val running: MutableMap<String, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val pending: MutableMap<String, MutableList<ImageDownloadRunnable>> = mutableMapOf()
private val lock = ReentrantLock()
private val condition = lock.newCondition()
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
fun init() {
coroutineScope.launch(Dispatchers.Default) {
val accepted: MutableList<ImageDownloadRunnable> = mutableListOf()
val candidates: MutableList<ImageDownloadRunnable> = mutableListOf()
while (isActive) {
lock.withLock {
candidates.addAll(getCandidates(candidateCount()))
candidates.forEach {
if (canRun(it.context.image.host)) {
accepted.add(it)
running[it.context.image.host]!!.add(it)
log.debug("${it.context.image.url} accepted to run")
}
}
accepted.forEach {
pending[it.context.image.host]?.remove(it)
scheduleForDownload(it)
}
accepted.clear()
candidates.clear()
try {
condition.await()
} catch (e: InterruptedException) {
Thread.currentThread().interrupt()
}
}
}
}
}
fun destroy() {
stop(dataTransaction.findAllPosts().map { it.postId })
}
fun stopAll(postIds: List<String>?) {
if (postIds != null) {
stop(postIds)
} else {
stop(dataTransaction.findAllPosts().map(Post::postId))
}
}
fun restartAll(postIds: List<String> = listOf()) {
if (postIds.isNotEmpty()) {
restart(postIds)
} else {
restart(dataTransaction.findAllPosts().map(Post::postId))
}
}
private fun restart(postIds: List<String>) {
lock.withLock {
val data: MutableMap<Post, Collection<Image>> = mutableMapOf()
for (postId in postIds) {
if (isPending(postId)) {
log.warn("Cannot restart, jobs are currently running for post id $postIds")
continue
}
val images: List<Image> =
dataTransaction.findByPostIdAndIsNotCompleted(postId)
if (images.isEmpty()) {
continue
}
val post: Post =
dataTransaction.findPostsByPostId(postId).orElseThrow()
log.debug("Restarting {} jobs for post id {}", images.size, postIds)
post.status = Status.PENDING
dataTransaction.update(post)
data[post] = images
}
for ((_, images) in data) {
for (image in images) {
log.debug("Enqueuing a job for ${image.url}")
with(image) {
this.status = Status.PENDING
this.current = 0
}
dataTransaction.update(image)
val imageDownloadRunnable = ImageDownloadRunnable(
image.id, settingsService.settings
)
pending.computeIfAbsent(
image.host
) { mutableListOf() }
pending[image.host]!!.add(imageDownloadRunnable)
}
}
condition.signal()
}
}
private fun isPending(postId: String): Boolean {
lock.withLock {
return pending.values.flatten().any { it.context.image.postId == postId }
}
}
private fun isRunning(postId: String): Boolean {
lock.withLock {
return running.values.flatten().any { it.context.image.postId == postId }
}
}
private fun stop(postIds: List<String>) {
lock.withLock {
for (postId in postIds) {
val post: Post =
dataTransaction.findPostsByPostId(postId).orElseThrow()
pending.values.forEach { pending ->
pending.removeIf { it.context.image.postId == postId }
}
running.values.flatten()
.filter { p: ImageDownloadRunnable -> p.context.image.postId == postId }
.forEach { obj: ImageDownloadRunnable -> obj.stop() }
dataTransaction.stopImagesByPostIdAndIsNotCompleted(postId)
dataTransaction.finishPost(post)
}
}
}
private fun canRun(host: String): Boolean {
val totalRunning = running.values.sumOf { it.size }
return (running[host]!!.size < settingsService.settings.connectionSettings.maxThreads && if (settingsService.settings.connectionSettings.maxTotalThreads == 0) totalRunning < maxPoolSize else totalRunning < settingsService.settings.connectionSettings.maxTotalThreads)
}
private fun candidateCount(): Map<String, Int> {
val map: MutableMap<String, Int> = mutableMapOf()
Host.getHosts().forEach { host: String ->
val imageDownloadRunnableList: List<ImageDownloadRunnable> = running.computeIfAbsent(
host
) { mutableListOf() }
val count: Int =
settingsService.settings.connectionSettings.maxThreads - imageDownloadRunnableList.size
log.debug("Download slots for $host: $count")
map[host] = count
}
return map
}
private fun getCandidates(candidateCount: Map<String, Int>): List<ImageDownloadRunnable> {
val hostIntegerMap: MutableMap<String, Int> = candidateCount.toMutableMap()
val candidates: MutableList<ImageDownloadRunnable> = mutableListOf()
hosts@ for (host in pending.keys) {
val list: List<ImageDownloadRunnable> =
pending[host]!!.sortedWith(Comparator.comparingInt<ImageDownloadRunnable> { it.context.post.rank }
.thenComparingInt { it.context.image.index })
for (imageDownloadRunnable in list) {
val count = hostIntegerMap[host] ?: 0
if (count > 0) {
candidates.add(imageDownloadRunnable)
hostIntegerMap[host] = count - 1
} else {
continue@hosts
}
}
}
if (log.isDebugEnabled) {
val collect: Map<String, List<ImageDownloadRunnable>> =
candidates.stream().collect(Collectors.groupingBy { it.context.image.host })
collect.forEach {
log.debug(
"Candidate download for ${it.key} ${it.value.size}/${candidateCount[it.key]}"
)
}
}
return candidates.sortedWith(Comparator.comparing { v: ImageDownloadRunnable -> v.context.post.rank })
}
private fun scheduleForDownload(imageDownloadRunnable: ImageDownloadRunnable) {
log.debug("Scheduling a job for ${imageDownloadRunnable.context.image.url}")
eventBus.publishEvent(QueueStateEvent(QueueState(runningCount(), pendingCount())))
Failsafe.with<Any, RetryPolicy<Any>>(retryPolicyService.buildRetryPolicyForDownload())
.onFailure {
try {
eventRepository.save(
LogEvent(
type = LogEvent.Type.DOWNLOAD,
status = LogEvent.Status.ERROR,
message = "Failed to download ${imageDownloadRunnable.context.image.url}\n ${it.failure.formatToString()}"
)
)
} catch (exp: Exception) {
log.error("Failed to save event", exp)
}
log.error(
"Failed to download ${imageDownloadRunnable.context.image.url} after ${it.attemptCount} tries",
it.failure
)
val image = imageDownloadRunnable.context.image
image.status = Status.ERROR
dataTransaction.update(image)
eventBus.publishEvent(ErrorCountEvent(ErrorCount(dataTransaction.countImagesInError())))
}.onComplete {
afterJobFinish(imageDownloadRunnable)
eventBus.publishEvent(
QueueStateEvent(
QueueState(
runningCount(),
pendingCount()
)
)
)
log.debug(
"Finished downloading ${imageDownloadRunnable.context.image.url}"
)
}.runAsync(imageDownloadRunnable::run)
}
private fun afterJobFinish(imageDownloadRunnable: ImageDownloadRunnable) {
lock.withLock {
val image = imageDownloadRunnable.context.image
running[image.host]!!.remove(imageDownloadRunnable)
if (!isPending(image.postId) && !isRunning(
image.postId
)
) {
dataTransaction.finishPost(imageDownloadRunnable.context.post)
}
condition.signal()
}
}
private fun pendingCount(): Int {
return pending.values.sumOf { it.size }
}
private fun runningCount(): Int {
return running.values.sumOf { it.size }
}
}
@@ -0,0 +1,22 @@
package me.mnlr.vripper.download
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.services.DataTransaction
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
class ImageDownloadContext(val imageId: Long, val settings: Settings) : KoinComponent {
private val dataTransaction: DataTransaction by inject()
val httpContext: HttpClientContext = HttpClientContext.create()
val postId = image.postIdRef
var stopped = false
val image: Image
get() = dataTransaction.findImageById(imageId).orElseThrow()
val post: Post
get() = dataTransaction.findPostById(postId).orElseThrow()
}
@@ -0,0 +1,188 @@
package me.mnlr.vripper.download
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.getExtension
import me.mnlr.vripper.host.DownloadedImage
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.host.ImageMimeType
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.services.*
import net.jodah.failsafe.function.CheckedRunnable
import org.apache.hc.client5.http.cookie.BasicCookieStore
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.io.IOException
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.StandardCopyOption
import java.util.*
import kotlin.io.path.pathString
class ImageDownloadRunnable(
private val imageInternalId: Long, private val settings: Settings
) : KoinComponent, CheckedRunnable {
private val log by LoggerDelegate()
private val dataTransaction: DataTransaction by inject()
private val hosts: List<Host> = getKoin().getAll()
val context: ImageDownloadContext = ImageDownloadContext(imageInternalId, settings)
private val image: Image
get() = context.image
private val post: Post
get() = context.post
private var stopped: Boolean
get() = context.stopped
set(value) {
context.stopped = value
}
private val httpContext: HttpClientContext
get() = context.httpContext
init {
httpContext.cookieStore = BasicCookieStore()
httpContext.setAttribute(
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES, HTTPService.ContextAttributes()
)
}
@Throws(DownloadException::class)
fun download() {
try {
val image = image
with(image) {
this.status = Status.DOWNLOADING
this.current = 0
}
dataTransaction.update(image)
synchronized(post.id.toString().intern()) {
val post = context.post
if (post.status != Status.DOWNLOADING) {
post.status = Status.DOWNLOADING
dataTransaction.update(post)
}
}
if (stopped) {
return
}
log.debug("Getting image url and name from ${image.url} using ${image.host}")
val host = hosts.first { it.isSupported(image.url) }
val downloadedImage = host.downloadInternal(image.url, context)
log.debug("Resolved name for ${image.url}: ${downloadedImage.name}")
log.debug(
"Downloaded image ${image.url} to ${downloadedImage.path}"
)
val sanitizedFileName = PathUtils.sanitize(downloadedImage.name)
log.debug(
"Sanitizing image name from ${downloadedImage.name} to $sanitizedFileName"
)
checkImageTypeAndRename(
context.post, downloadedImage, image.index
)
} catch (e: Exception) {
if (stopped) {
return
}
throw DownloadException(e)
} finally {
synchronized(post.id.toString().intern()) {
val image = image
if (image.current == image.total && image.total > 0) {
image.status = Status.FINISHED
val post = context.post
post.done += 1
dataTransaction.update(post)
} else if (stopped) {
image.status = Status.STOPPED
} else {
image.status = Status.ERROR
}
dataTransaction.update(image)
}
}
}
@Throws(HostException::class)
private fun checkImageTypeAndRename(
post: Post, downloadedImage: DownloadedImage, index: Int
) {
val existingExtension = getExtension(downloadedImage.name)
val extension = when (downloadedImage.type) {
ImageMimeType.IMAGE_BMP -> "BMP"
ImageMimeType.IMAGE_GIF -> "GIF"
ImageMimeType.IMAGE_JPEG -> "JPG"
ImageMimeType.IMAGE_PNG -> "PNG"
ImageMimeType.IMAGE_WEBP -> "WEBP"
}
val filename =
if (existingExtension.isBlank()) "${downloadedImage.name}.$extension" else downloadedImage.name
try {
val downloadDestinationFolder = Path.of(post.downloadDirectory)
synchronized(downloadDestinationFolder.pathString.intern()) {
Files.createDirectories(downloadDestinationFolder)
}
val image = downloadDestinationFolder.resolve(
"${
if (settings.downloadSettings.forceOrder) String.format(
"%03d_", index + 1
) else ""
}$filename"
)
Files.copy(downloadedImage.path, image, StandardCopyOption.REPLACE_EXISTING)
} catch (e: Exception) {
throw HostException("Failed to rename the image", e)
} finally {
try {
Files.delete(downloadedImage.path)
} catch (e: IOException) {
log.warn(
"Failed to delete temporary file ${downloadedImage.path}"
)
}
}
}
@Throws(Exception::class)
override fun run() {
if (stopped) {
return
}
download()
}
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (other == null || javaClass != other.javaClass) return false
val that = other as ImageDownloadRunnable
return imageInternalId == that.imageInternalId
}
override fun hashCode(): Int {
return Objects.hash(imageInternalId)
}
fun stop() {
stopped = true
val contextAttributes = httpContext.getAttribute(
HTTPService.ContextAttributes.CONTEXT_ATTRIBUTES,
HTTPService.ContextAttributes::class.java
)
if (contextAttributes != null) {
synchronized(contextAttributes.requests) {
for (request in contextAttributes.requests) {
request.abort()
}
}
}
}
}
@@ -0,0 +1,158 @@
package me.mnlr.vripper.download
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.entities.LogEvent.Status.*
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.PostParseException
import me.mnlr.vripper.formatToString
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.parser.ThreadLookupAPIResponseHandler
import me.mnlr.vripper.repositories.LogEventRepository
import me.mnlr.vripper.services.*
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.net.URIBuilder
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.net.URISyntaxException
import java.util.*
import javax.xml.parsers.SAXParserFactory
class PostDownloadRunnable(private val threadId: String, private val postId: String) : KoinComponent, Runnable {
private val log by LoggerDelegate()
private val dataTransaction: DataTransaction by inject()
private val settingsService: SettingsService by inject()
private val vgAuthService: VGAuthService by inject()
private val eventRepository: LogEventRepository by inject()
private val cm: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val downloadService: DownloadService by inject()
private val link: String =
"${settingsService.settings.viperSettings.host}/threads/$threadId?p=$postId"
private val logEvent: LogEvent
init {
logEvent = eventRepository.save(
LogEvent(
type = LogEvent.Type.POST,
status = PENDING,
message = "Processing $link"
)
)
}
override fun run() {
try {
eventRepository.update(logEvent.copy(status = PROCESSING))
if (dataTransaction.exists(postId)) {
log.warn(String.format("skipping %s, already loaded", postId))
eventRepository.update(
logEvent.copy(
status = ERROR,
message = String.format("Gallery %s is already loaded", link)
)
)
return
}
val postItem: PostItem = try {
parse()
} catch (e: PostParseException) {
val error = String.format("parsing failed for gallery %s", link)
log.error(error, e)
eventRepository.update(
logEvent.copy(
status = ERROR, message = """
$error
${e.formatToString()}
""".trimIndent()
)
)
return
}
if (postItem.imageItemList.isEmpty()) {
val error = String.format("Gallery %s contains no images to download", link)
log.error(error)
eventRepository.update(logEvent.copy(status = ERROR, message = error))
return
}
val post = dataTransaction.newPost(postItem)
vgAuthService.leaveThanks(post)
// metadataService.startFetchingMetadata(post)
if (settingsService.settings.downloadSettings.autoStart) {
log.debug("Auto start downloads option is enabled")
downloadService.restartAll(listOf(postItem.postId))
log.debug(String.format("Done enqueuing jobs for %s", postItem.url))
}
eventRepository.update(
logEvent.copy(
status = DONE,
message = String.format(
"Gallery %s is successfully added to download queue",
link
)
)
)
} catch (e: Exception) {
val error = String.format("Error when adding gallery %s", link)
log.error(error, e)
eventRepository.update(
logEvent.copy(
status = ERROR, message = """
$error
${e.formatToString()}
""".trimIndent()
)
)
}
}
@Throws(PostParseException::class)
fun parse(): PostItem {
log.debug("Parsing post $postId")
val httpGet: HttpGet = try {
val uriBuilder = URIBuilder("${settingsService.settings.viperSettings.host}/vr.php")
uriBuilder.setParameter("p", postId)
cm.buildHttpGet(uriBuilder.build(), HttpClientContext.create())
} catch (e: URISyntaxException) {
throw PostParseException(e)
}
val threadLookupAPIResponseHandler = ThreadLookupAPIResponseHandler()
log.debug("Requesting $httpGet")
return try {
Failsafe.with(retryPolicyService.buildGenericRetryPolicy<Any>()).onFailure {
log.error(
"parsing failed for thread $threadId, post $postId", it.failure
)
}.get(CheckedSupplier {
val connection: HttpClient = cm.clientBuilder.build()
(connection.execute(
httpGet, vgAuthService.context
) as CloseableHttpResponse).use { response ->
try {
if (response.code / 100 != 2) {
throw DownloadException("Unexpected response code '${response.code}' for $httpGet")
}
factory.newSAXParser()
.parse(response.entity.content, threadLookupAPIResponseHandler)
threadLookupAPIResponseHandler.result.postItemList.first()
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
})
} catch (e: Exception) {
throw PostParseException(e)
}
}
companion object {
private val factory = SAXParserFactory.newInstance()
}
}
@@ -0,0 +1,31 @@
package me.mnlr.vripper.entities
import me.mnlr.vripper.entities.domain.Status
data class Image(
var id: Long = -1,
val postId: String,
val url: String,
val thumbUrl: String,
val host: String,
val index: Int,
val postIdRef: Long,
var total: Long = -1,
var current: Long = 0,
var status: Status = Status.STOPPED,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as Image
if (url != other.url) return false
return true
}
override fun hashCode(): Int {
return url.hashCode()
}
}
@@ -0,0 +1,23 @@
package me.mnlr.vripper.entities
import com.fasterxml.jackson.annotation.JsonFormat
import java.time.LocalDateTime
data class LogEvent(
val id: Long? = null,
val type: Type,
val status: Status,
@JsonFormat(pattern="yyyy-MM-dd HH:mm:ss") val time: LocalDateTime = LocalDateTime.now(),
val message: String,
) {
enum class Type(val stringValue: String) {
POST("Post"), THREAD("Thread"), THANKS("Thanks"), METADATA("Metadata"), SCAN("Scan"), DOWNLOAD(
"Download"
)
}
enum class Status(val stringValue: String) {
PENDING("Pending"), PROCESSING("Processing"), DONE("Done"), ERROR("Error")
}
}
@@ -0,0 +1,19 @@
package me.mnlr.vripper.entities
class Metadata {
var postIdRef: Long? = null
var postId: String? = null
var postedBy: String? = null
var resolvedNames = emptyList<String>()
companion object {
fun from(metadata: Metadata): Metadata {
val copy = Metadata()
copy.postIdRef = metadata.postIdRef
copy.postId = metadata.postId
copy.postedBy = metadata.postedBy
copy.resolvedNames = metadata.resolvedNames
return copy
}
}
}
@@ -0,0 +1,38 @@
package me.mnlr.vripper.entities
import com.fasterxml.jackson.annotation.JsonFormat
import me.mnlr.vripper.entities.domain.Status
import java.time.LocalDateTime
data class Post(
var id: Long = -1,
val postTitle: String,
val threadTitle: String,
val forum: String,
val url: String,
val token: String,
val postId: String,
val threadId: String,
val total: Int,
val hosts: Set<String>,
val downloadDirectory: String,
@JsonFormat(pattern="yyyy-MM-dd HH:mm:ss") val addedOn: LocalDateTime = LocalDateTime.now(),
var status: Status = Status.STOPPED,
var done: Int = 0,
var rank: Int = Int.MAX_VALUE
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as Post
if (postId != other.postId) return false
return true
}
override fun hashCode(): Int {
return postId.hashCode()
}
}
@@ -0,0 +1,22 @@
package me.mnlr.vripper.entities
data class Thread(
val id: Long? = null,
val title: String,
val link: String,
val threadId: String,
var total: Int = 0,
) {
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as Thread
return threadId == other.threadId
}
override fun hashCode(): Int {
return threadId.hashCode()
}
}
@@ -0,0 +1,5 @@
package me.mnlr.vripper.entities.domain
enum class Status(val stringValue: String) {
PENDING("Pending"), DOWNLOADING("Downloading"), FINISHED("Finished"), ERROR("Error"), STOPPED("Stopped")
}
@@ -0,0 +1,23 @@
package me.mnlr.vripper.event
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.model.DownloadSpeed
import me.mnlr.vripper.model.ErrorCount
import me.mnlr.vripper.model.QueueState
import me.mnlr.vripper.model.Settings
data class PostCreateEvent(val post: Post)
data class PostUpdateEvent(val post: Post)
data class PostDeleteEvent(val postId: String)
data class ImageCreateEvent(val image: Image)
data class ImageUpdateEvent(val image: Image)
data class ThreadCreateEvent(val thread: Thread)
data class ThreadDeleteEvent(val threadId: String)
class ThreadClearEvent
data class VGUserLoginEvent(val username: String)
data class DownloadSpeedEvent(val downloadSpeed: DownloadSpeed)
data class QueueStateEvent(val queueState: QueueState)
data class ErrorCountEvent(val errorCount: ErrorCount)
data class SettingsUpdateEvent(val settings: Settings)
@@ -0,0 +1,29 @@
package me.mnlr.vripper.event
import kotlinx.coroutines.channels.BufferOverflow
import kotlinx.coroutines.ensureActive
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.asSharedFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.flow.filterIsInstance
import kotlin.coroutines.coroutineContext
object EventBus {
private val _events = MutableSharedFlow<Any>(
onBufferOverflow = BufferOverflow.DROP_OLDEST,
extraBufferCapacity = 100_000
)
val events = _events.asSharedFlow()
fun publishEvent(event: Any) {
_events.tryEmit(event)
}
suspend inline fun <reified T> subscribe(crossinline onEvent: (T) -> Unit) {
events.filterIsInstance<T>()
.collectLatest { event ->
coroutineContext.ensureActive()
onEvent(event)
}
}
}
@@ -0,0 +1,6 @@
package me.mnlr.vripper.exception
class DownloadException : Exception {
constructor(message: String?) : super(message)
constructor(e: Throwable?) : super(e)
}
@@ -0,0 +1,7 @@
package me.mnlr.vripper.exception
class HostException : Exception {
constructor(e: Throwable?) : super(e)
constructor(message: String?) : super(message)
constructor(message: String?, e: Throwable?) : super(message, e)
}
@@ -0,0 +1,3 @@
package me.mnlr.vripper.exception
class HtmlProcessorException(e: Throwable?) : Exception(e)
@@ -0,0 +1,7 @@
package me.mnlr.vripper.exception
class PostParseException : Exception {
constructor(message: String?) : super(message)
constructor(message: String?, e: Throwable?) : super(message, e)
constructor(e: Throwable?) : super(e)
}
@@ -0,0 +1,7 @@
package me.mnlr.vripper.exception
class QueueException : Exception {
constructor(message: String?) : super(message)
constructor(message: String?, e: Throwable?) : super(message, e)
constructor(e: Throwable?) : super(e)
}
@@ -0,0 +1,3 @@
package me.mnlr.vripper.exception
class RenameException(message: String?, e: Exception?) : Exception(message, e)
@@ -0,0 +1,3 @@
package me.mnlr.vripper.exception
class ValidationException(message: String) : Exception(message)
@@ -0,0 +1,6 @@
package me.mnlr.vripper.exception
class VripperException : Exception {
constructor(message: String?) : super(message)
constructor(e: Throwable?) : super(e)
}
@@ -0,0 +1,3 @@
package me.mnlr.vripper.exception
class XpathException(e: Throwable?) : Exception(e)
@@ -0,0 +1,90 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import org.w3c.dom.Document
import org.w3c.dom.Node
class AcidimgHost(
private val httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("acidimg.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String, document: Document, context: ImageDownloadContext
): Pair<String, String> {
try {
log.debug(
String.format(
"Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url
)
)
XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
} catch (e: XpathException) {
throw HostException(e)
}
log.debug(String.format("Click button found for %s", url))
val client: HttpClient = httpService.clientBuilder.build()
val httpPost = httpService.buildHttpPost(url, context.httpContext)
httpPost.addHeader("Referer", url)
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("imgContinue", "Continue to your image"))
try {
httpPost.entity = UrlEncodedFormEntity(params)
} catch (e: Exception) {
throw HostException(e)
}
log.debug(String.format("Requesting %s", httpPost))
val doc = try {
(client.execute(
httpPost, context.httpContext
) as CloseableHttpResponse).use { response ->
log.debug(String.format("Cleaning response for %s", httpPost))
try {
HtmlProcessorService.clean(response.entity.content)
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
} catch (e: Exception) {
throw HostException(e)
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'", IMG_XPATH, url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim()
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim()
Pair(
imgTitle.ifEmpty { getDefaultImageName(imgUrl) }, imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val CONTINUE_BUTTON_XPATH = "//input[@id='continuebutton']"
private const val IMG_XPATH = "//img[@class='centred']"
}
}
@@ -0,0 +1,61 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
class DPicMeHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("dpic.me", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle =
Optional.ofNullable(imgNode.attributes.getNamedItem("alt"))
.map { e: Node -> e.textContent.trim() }
.orElse("")
val imgUrl =
Optional.ofNullable(imgNode.attributes.getNamedItem("src"))
.map { e: Node -> e.textContent.trim() }
.orElse("")
var defaultName: String = UUID.randomUUID().toString()
val index = imgUrl.lastIndexOf('/')
if (index != -1 && index < imgUrl.length) {
defaultName = imgUrl.substring(imgUrl.lastIndexOf('/') + 1)
}
Pair((imgTitle.ifEmpty { defaultName })!!, imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='pic']"
}
}
@@ -0,0 +1,239 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.getFileNameWithoutExtension
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.Header
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.w3c.dom.Document
import java.nio.file.Files
import java.nio.file.Path
abstract class Host(
val hostId: String,
private val httpService: HTTPService,
private val dataTransaction: DataTransaction,
private val downloadSpeedService: DownloadSpeedService
) {
private val log by LoggerDelegate()
companion object {
private const val READ_BUFFER_SIZE = 8192
private val hosts: MutableList<String> = mutableListOf()
fun getHosts(): List<String> {
return hosts.toList()
}
}
init {
hosts.add(hostId)
}
@Throws(HostException::class)
abstract fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String>
@Throws(HostException::class)
fun downloadInternal(url: String, context: ImageDownloadContext): DownloadedImage {
val headers = head(url, context.httpContext)
// is the body of type image ?
val imageMimeType = getImageMimeType(headers)
val downloadedImage = if (imageMimeType != null) {
// a direct link, awesome
val downloadedImage = fetch(url, context) {
handleImageDownload(it, context)
}
DownloadedImage(getDefaultImageName(url), downloadedImage.first, downloadedImage.second)
} else {
// linked image ?
val value = headers.find { it.name.contains("content-type", true) }?.value
if (value != null) {
if (value.contains("text/html")) {
val document = fetch(url, context) {
HtmlProcessorService.clean(it.entity.content)
}
if (log.isDebugEnabled) {
log.debug("Cleaning $url response", url)
}
val resolvedImage = resolve(url, document, context)
val downloadImage: Pair<Path, ImageMimeType> =
fetch(resolvedImage.second, context) {
handleImageDownload(it, context)
}
DownloadedImage(resolvedImage.first, downloadImage.first, downloadImage.second)
} else {
throw HostException("Unable to download $url, can't process content type $value")
}
} else {
throw HostException("Unexpected server response for $url, response have no content type")
}
}
return downloadedImage
}
private fun handleImageDownload(
response: CloseableHttpResponse,
context: ImageDownloadContext
): Pair<Path, ImageMimeType> {
val mimeType = getImageMimeType(response.headers)
?: throw HostException("Unsupported image type ${response.getFirstHeader("content-type")}")
val tempImage = Files.createTempFile(Path.of(context.settings.downloadSettings.tempPath), "vripper_", ".tmp")
return response.entity.content.use { inputStream ->
try {
Files.newOutputStream(tempImage).use { fos ->
val image = context.image
image.total = response.entity.contentLength
dataTransaction.update(image)
log.debug(
"Length is ${image.total}"
)
log.debug(
"Starting data transfer"
)
val buffer = ByteArray(READ_BUFFER_SIZE)
var read: Int
while (inputStream.read(buffer, 0, READ_BUFFER_SIZE)
.also { read = it } != -1 && !context.stopped
) {
fos.write(buffer, 0, read)
with(image) {
current += read
}
downloadSpeedService.reportDownloadedBytes(read.toLong())
dataTransaction.update(image)
}
Pair(tempImage, mimeType)
}
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
}
fun isSupported(url: String): Boolean {
return url.contains(hostId)
}
@Throws(HostException::class)
fun head(url: String, context: HttpClientContext): Array<Header> {
val client: CloseableHttpAsyncClient = httpService.clientBuilder.build()
val httpGet = httpService.buildHttpHead(url, context)
log.debug(String.format("Requesting %s", url))
return try {
(client.execute(
httpGet,
context
) as CloseableHttpResponse).use { response ->
try {
if (response.code / 100 != 2) {
throw HostException(
String.format(
"Unexpected response code: %d", response.code
)
)
}
response.headers
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
} catch (e: Exception) {
throw HostException(e)
}
}
@Throws(HostException::class)
fun <T> fetch(
url: String,
context: ImageDownloadContext,
transformer: (CloseableHttpResponse) -> T
): T {
val client: HttpClient = httpService.clientBuilder.build()
val httpGet = httpService.buildHttpGet(url, context.httpContext)
httpGet.addHeader("Referer", context.image.url)
log.debug(String.format("Requesting %s", url))
return try {
(client.execute(
httpGet,
context.httpContext
) as CloseableHttpResponse).use { response ->
if (response.code / 100 != 2) {
EntityUtils.consumeQuietly(response.entity)
throw DownloadException(
"Server returned code ${response.code}"
)
}
try {
transformer(response)
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
} catch (e: Exception) {
throw HostException(e)
}
}
private fun getImageMimeType(headers: Array<Header>): ImageMimeType? {
// first check if content type header exists
val value = headers.find { it.name.contains("content-type", true) }?.value
// header found, check the type
return if (value != null) {
ImageMimeType.values().find {
value.contains(it.strValue, true)
}
} else {
null
}
}
fun getDefaultImageName(imgUrl: String): String {
val imageTitle = imgUrl.substring(imgUrl.lastIndexOf('/') + 1)
log.debug(String.format("Extracting name from url %s: %s", imgUrl, imageTitle))
return getFileNameWithoutExtension(imageTitle)
}
override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false
other as Host
if (hostId != other.hostId) return false
return true
}
override fun hashCode(): Int {
return hostId.hashCode()
}
override fun toString(): String {
return hostId
}
}
data class DownloadedImage(val name: String, val path: Path, val type: ImageMimeType)
enum class ImageMimeType(val strValue: String) {
IMAGE_BMP("image/bmp"),
IMAGE_GIF("image/gif"),
IMAGE_JPEG("image/jpeg"),
IMAGE_PNG("image/png"),
IMAGE_WEBP("image/webp"),
}
@@ -0,0 +1,84 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.impl.cookie.BasicClientCookie
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.sql.Date
import java.time.LocalDateTime
import java.time.ZoneId
import java.util.*
class ImageBamHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imagebam.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val doc = try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_XPATH, url))
if (XpathService.getAsNode(document, CONTINUE_XPATH) != null) {
val clientCookie = BasicClientCookie("nsfw_inter", "1")
clientCookie.domain = "www.imagebam.com"
clientCookie.path = "/"
clientCookie.expiryDate =
Date.from(
LocalDateTime.now().plusDays(3).atZone(ZoneId.systemDefault()).toInstant()
)
context.httpContext.cookieStore.addCookie(clientCookie)
fetch(url, context) {
HtmlProcessorService.clean(it.entity.content)
}
} else {
document
}
} catch (e: XpathException) {
throw HostException(e)
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = Optional.ofNullable(imgNode.attributes.getNamedItem("alt"))
.map { e: Node -> e.textContent.trim { it <= ' ' } }
.orElse("")
val imgUrl = Optional.ofNullable(imgNode.attributes.getNamedItem("src"))
.map { e: Node -> e.textContent.trim { it <= ' ' } }
.orElse("")
var defaultName: String = UUID.randomUUID().toString()
val index = imgUrl.lastIndexOf('/')
if (index != -1 && index < imgUrl.length) {
defaultName = imgUrl.substring(imgUrl.lastIndexOf('/') + 1)
}
Pair((imgTitle.ifEmpty { defaultName })!!, imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[contains(@class,'main-image')]"
private const val CONTINUE_XPATH = "//*[contains(text(), 'Continue')]"
}
}
@@ -0,0 +1,53 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
class ImageTwistHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imagetwist.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle =
Optional.ofNullable(imgNode.attributes.getNamedItem("alt"))
.map { obj: Node -> obj.textContent }
.map { obj: String -> obj.trim { it <= ' ' } }.orElse(null)
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(imgTitle!!, imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[contains(@class, 'img')]"
}
}
@@ -0,0 +1,72 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class ImageVenueHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imagevenue.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val doc = try {
log.debug(
String.format(
"Looking for xpath expression %s in %s",
CONTINUE_BUTTON_XPATH,
url
)
)
if (XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH) != null) {
// Button detected. No need to actually click it, just make the call again.
fetch(url, context) {
HtmlProcessorService.clean(it.entity.content)
}
} else {
document
}
} catch (e: XpathException) {
throw HostException(e)
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(
if (imgTitle.isEmpty()) imgUrl.substring(imgUrl.lastIndexOf('/') + 1) else imgTitle,
imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val CONTINUE_BUTTON_XPATH = "//a[@title='Continue to ImageVenue']"
private const val IMG_XPATH = "//a[@data-toggle='full']/img"
}
}
@@ -0,0 +1,51 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
class ImageZillaHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imagezilla.net", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
log.debug(String.format("Resolving name for %s", url))
var title = titleNode.attributes.getNamedItem("title").textContent.trim()
titleNode.textContent.trim()
if (title.isEmpty()) {
title = getDefaultImageName(url)
}
return try {
Pair(title, url.replace("show", "images"))
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='photo']"
}
}
@@ -0,0 +1,51 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class ImgSpiceHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imgspice.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(
imgTitle.ifEmpty { imgUrl.substring(imgUrl.lastIndexOf('/') + 1) }, imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='imgpreview']"
}
}
@@ -0,0 +1,49 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class ImgboxHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imgbox.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = imgNode.attributes.getNamedItem("title").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(imgTitle, imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='img']"
}
}
@@ -0,0 +1,100 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.HtmlProcessorException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.io.IOException
class ImxHost(
private val httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("imx.to", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val httpsUrl = url.replace("http:", "https:")
var value: String? = null
try {
log.debug("Looking for xpath expression $CONTINUE_BUTTON_XPATH in $httpsUrl")
val contDiv = XpathService.getAsNode(document, CONTINUE_BUTTON_XPATH)
?: throw HostException("$CONTINUE_BUTTON_XPATH cannot be found")
val node = contDiv.attributes.getNamedItem("value")
if (node != null) {
value = node.textContent
}
} catch (e: XpathException) {
throw HostException(e)
}
if (value == null) {
throw HostException("Failed to obtain value attribute from continue input")
}
log.debug("Click button found for $httpsUrl")
val client: HttpClient = httpService.clientBuilder.build()
val httpPost: HttpPost = httpService.buildHttpPost(httpsUrl, context.httpContext)
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("imgContinue", value))
try {
httpPost.entity = UrlEncodedFormEntity(params)
} catch (e: Exception) {
throw HostException(e)
}
log.debug("Requesting $httpPost")
val doc = try {
(client.execute(
httpPost, context.httpContext
) as CloseableHttpResponse).use { response ->
log.debug("Cleaning response for $httpPost")
try {
HtmlProcessorService.clean(response.entity.content)
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
} catch (e: IOException) {
throw HostException(e)
} catch (e: HtmlProcessorException) {
throw HostException(e)
}
val imgNode: Node = try {
log.debug("Looking for xpath expression $IMG_XPATH in $httpsUrl")
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
"Xpath $IMG_XPATH cannot be found in $httpsUrl"
)
return try {
log.debug("Resolving name and image url for $httpsUrl")
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(
imgTitle.ifEmpty { getDefaultImageName(imgUrl) }, imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val CONTINUE_BUTTON_XPATH = "//*[@name='imgContinue']"
private const val IMG_XPATH = "//img[@class='centred']"
}
}
@@ -0,0 +1,77 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.net.URI
import java.net.URISyntaxException
class PimpandhostHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("pimpandhost.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val newUrl: String
try {
newUrl = appendUri(url.replace("http://", "https://").replace("-medium(\\.html)?".toRegex(), ""), "size=original")
} catch (e: Exception) {
throw HostException(e)
}
val doc = fetch(newUrl, context) {
HtmlProcessorService.clean(it.entity.content)
}
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, newUrl))
XpathService.getAsNode(doc, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", newUrl))
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl =
"https:" + imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(
imgTitle.ifEmpty { imgUrl.substring(imgUrl.lastIndexOf('/') + 1) }, imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
@Throws(URISyntaxException::class)
fun appendUri(uri: String, appendQuery: String): String {
val oldUri = URI(uri)
var newQuery = oldUri.query
if (newQuery == null) {
newQuery = appendQuery
} else {
newQuery += "&$appendQuery"
}
return URI(
oldUri.scheme, oldUri.authority, oldUri.path, newQuery, oldUri.fragment
).toString()
}
companion object {
private const val IMG_XPATH = "//img[contains(@class, 'original')]"
}
}
@@ -0,0 +1,48 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class PixRouteHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("pixroute.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
Pair(imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' },
imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' })
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='imgpreview']"
}
}
@@ -0,0 +1,49 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class PixhostHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("pixhost.to", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = imgNode.attributes.getNamedItem("alt").textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' }
Pair(imgTitle.substring(imgTitle.indexOf('_') + 1), imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[@id='image']"
}
}
@@ -0,0 +1,63 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
class PixxxelsHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("pixxxels.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
XpathService.getAsNode(document, TITLE_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
TITLE_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = titleNode.textContent.trim { it <= ' ' }
val imgUrl = imgNode.attributes.getNamedItem("href").textContent.trim { it <= ' ' }
Pair(
imgTitle.ifEmpty { imgUrl.substring(imgUrl.lastIndexOf('/') + 1) }, imgUrl
)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//*[@id='download']"
private const val TITLE_XPATH = "//*[contains(@class,'imagename')]"
}
}
@@ -0,0 +1,64 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
class PostImgHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("postimg.cc", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val titleNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
XpathService.getAsNode(document, TITLE_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
TITLE_XPATH,
url
)
)
val urlNode = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle = Optional.ofNullable(titleNode)
.map { node: Node -> node.textContent.trim { it <= ' ' } }
.orElseGet { getDefaultImageName(url) }
Pair(imgTitle, urlNode.attributes.getNamedItem("href").textContent.trim { it <= ' ' })
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val TITLE_XPATH = "//span[contains(@class,'imagename')]"
private const val IMG_XPATH = "//a[@id='download']"
}
}
@@ -0,0 +1,51 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
class TurboImageHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("turboimagehost.com", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
var title: String?
title = try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url))
val titleNode: Node? = XpathService.getAsNode(document, TITLE_XPATH)
log.debug(String.format("Resolving name for %s", url))
titleNode?.textContent?.trim { it <= ' ' }
} catch (e: XpathException) {
throw HostException(e)
}
if (title.isNullOrEmpty()) {
title = getDefaultImageName(url)
}
val urlNode: Node = XpathService.getAsNode(document, IMG_XPATH)
?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return Pair(title, urlNode.attributes.getNamedItem("src").textContent.trim { it <= ' ' })
}
companion object {
private const val TITLE_XPATH = "//div[contains(@class,'titleFullS')]/h1"
private const val IMG_XPATH = "//img[@id='imageid']"
}
}
@@ -0,0 +1,53 @@
package me.mnlr.vripper.host
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.download.ImageDownloadContext
import me.mnlr.vripper.exception.HostException
import me.mnlr.vripper.exception.XpathException
import me.mnlr.vripper.services.*
import org.w3c.dom.Document
import org.w3c.dom.Node
import java.util.*
class ViprImHost(
httpService: HTTPService,
dataTransaction: DataTransaction,
downloadSpeedService: DownloadSpeedService,
) : Host("vipr.im", httpService, dataTransaction, downloadSpeedService) {
private val log by LoggerDelegate()
@Throws(HostException::class)
override fun resolve(
url: String,
document: Document,
context: ImageDownloadContext
): Pair<String, String> {
val imgNode: Node = try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url))
XpathService.getAsNode(document, IMG_XPATH)
} catch (e: XpathException) {
throw HostException(e)
} ?: throw HostException(
String.format(
"Xpath '%s' cannot be found in '%s'",
IMG_XPATH,
url
)
)
return try {
log.debug(String.format("Resolving name and image url for %s", url))
val imgTitle =
Optional.ofNullable(imgNode.attributes.getNamedItem("alt"))
.map { obj: Node -> obj.textContent }
.map { obj: String -> obj.trim() }.orElse(null)
val imgUrl = imgNode.attributes.getNamedItem("src").textContent.trim()
Pair(imgTitle!!, imgUrl)
} catch (e: Exception) {
throw HostException("Unexpected error occurred", e)
}
}
companion object {
private const val IMG_XPATH = "//img[contains(@class, 'img')]"
}
}
@@ -0,0 +1,32 @@
package me.mnlr.vripper.listeners
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import java.io.RandomAccessFile
import java.nio.file.Files
import kotlin.io.path.Path
import kotlin.io.path.pathString
import kotlin.system.exitProcess
object AppLock {
fun exclusiveLock() {
val lock = Path(baseDir).resolve(BASE_DIR_NAME).resolve("lock")
try {
val randomFile = RandomAccessFile(lock.pathString, "rw")
val channel = randomFile.channel
val fileLock = channel.tryLock()
if (fileLock == null) {
System.err.println("Another instance is already running in ${lock.parent.pathString}")
exitProcess(-1)
} else {
Runtime.getRuntime().addShutdownHook(Thread {
fileLock.release()
channel.close()
Files.deleteIfExists(lock)
})
}
} catch (e: Exception) {
println(e.toString())
}
}
}
@@ -0,0 +1,25 @@
package me.mnlr.vripper.listeners
import me.mnlr.vripper.download.DownloadService
import me.mnlr.vripper.services.*
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
open class OnStartupListener : KoinComponent {
private val dataTransaction: DataTransaction by inject()
private val downloadService: DownloadService by inject()
private val httpService: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val vgAuthService: VGAuthService by inject()
private val downloadSpeedService: DownloadSpeedService by inject()
open fun run() {
dataTransaction.setDownloadingToStopped()
dataTransaction.sortPostsByRank()
httpService.init()
retryPolicyService.init()
vgAuthService.init()
downloadService.init()
downloadSpeedService.init()
}
}
@@ -0,0 +1,3 @@
package me.mnlr.vripper.model
data class DownloadSpeed(val speed: String)
@@ -0,0 +1,3 @@
package me.mnlr.vripper.model
data class ErrorCount(val count: Int)
@@ -0,0 +1,5 @@
package me.mnlr.vripper.model
import me.mnlr.vripper.host.Host
data class ImageItem(val mainLink: String, val thumbLink: String, val host: Host)
@@ -0,0 +1,3 @@
package me.mnlr.vripper.model
data class LoggedUser(val user: String)
@@ -0,0 +1,15 @@
package me.mnlr.vripper.model
data class PostItem(
val threadId: String,
val threadTitle: String,
val postId: String,
val number: Int,
val title: String,
val imageCount: Int,
val url: String,
val hosts: List<Pair<String, Int>>,
val securityToken: String,
val forum: String,
val imageItemList: List<ImageItem>
)
@@ -0,0 +1,6 @@
package me.mnlr.vripper.model
data class QueueState(
val running: Int,
val remaining: Int
)
@@ -0,0 +1,38 @@
package me.mnlr.vripper.model
data class Settings(
var maxEventLog: Int = 1_000,
var connectionSettings: ConnectionSettings = ConnectionSettings(),
var downloadSettings: DownloadSettings = DownloadSettings(),
var viperSettings: ViperSettings = ViperSettings(),
var clipboardSettings: ClipboardSettings = ClipboardSettings()
)
data class ViperSettings(
var login: Boolean = false,
var username: String = "",
var password: String = "",
var thanks: Boolean = false,
var host: String = "https://vipergirls.to",
)
data class DownloadSettings(
var downloadPath: String = System.getProperty("user.home"),
var tempPath: String = System.getProperty("java.io.tmpdir"),
var autoStart: Boolean = true,
var autoQueueThreshold: Int = 1,
var forceOrder: Boolean = false,
var forumSubfolder: Boolean = false,
var threadSubLocation: Boolean = false,
var clearCompleted: Boolean = false,
var appendPostId: Boolean = false
)
data class ConnectionSettings(
var maxThreads: Int = 2,
var maxTotalThreads: Int = 0,
var timeout: Long = 30,
var maxAttempts: Int = 3,
)
data class ClipboardSettings(var enable: Boolean = false, var pollingRate: Int = 500)
@@ -0,0 +1,9 @@
package me.mnlr.vripper.model
data class ThreadItem(
val threadId: String,
val title: String,
val securityToken: String,
val forum: String,
val postItemList: List<PostItem>
)
@@ -0,0 +1,78 @@
package me.mnlr.vripper.parser
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.exception.DownloadException
import me.mnlr.vripper.exception.PostParseException
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.services.HTTPService
import me.mnlr.vripper.services.RetryPolicyService
import me.mnlr.vripper.services.SettingsService
import me.mnlr.vripper.services.VGAuthService
import net.jodah.failsafe.Failsafe
import net.jodah.failsafe.function.CheckedSupplier
import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.net.URIBuilder
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import java.io.BufferedInputStream
import java.net.URISyntaxException
import java.util.*
import javax.xml.parsers.SAXParserFactory
class ThreadLookupAPIParser(private val threadId: String) : KoinComponent {
private val log by LoggerDelegate()
private val cm: HTTPService by inject()
private val retryPolicyService: RetryPolicyService by inject()
private val vgAuthService: VGAuthService by inject()
private val settingsService: SettingsService by inject()
@Throws(PostParseException::class)
fun parse(): ThreadItem {
log.debug("Parsing thread $threadId")
val httpGet: HttpGet = try {
val uriBuilder = URIBuilder(settingsService.settings.viperSettings.host + "/vr.php")
uriBuilder.setParameter("t", threadId)
cm.buildHttpGet(uriBuilder.build(), HttpClientContext.create())
} catch (e: URISyntaxException) {
throw PostParseException(e)
}
val threadLookupAPIResponseHandler = ThreadLookupAPIResponseHandler()
log.debug("Requesting $httpGet")
return try {
Failsafe.with(retryPolicyService.buildGenericRetryPolicy<Any>()).onFailure {
log.error(
"parsing failed for thread $threadId",
it.failure
)
}.get(CheckedSupplier {
val connection: HttpClient = cm.clientBuilder.build()
(connection.execute(
httpGet, vgAuthService.context
) as CloseableHttpResponse).use { response ->
try {
if (response.code / 100 != 2) {
throw DownloadException("Unexpected response code '${response.code}' for $httpGet")
}
factory.newSAXParser().parse(
BufferedInputStream(response.entity.content),
threadLookupAPIResponseHandler
)
threadLookupAPIResponseHandler.result
} finally {
EntityUtils.consumeQuietly(response.entity)
}
}
})
} catch (e: Exception) {
throw PostParseException(e)
}
}
companion object {
private val factory = SAXParserFactory.newInstance()
}
}
@@ -0,0 +1,103 @@
package me.mnlr.vripper.parser
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.host.Host
import me.mnlr.vripper.model.ImageItem
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.services.SettingsService
import org.koin.core.component.KoinComponent
import org.koin.core.component.inject
import org.xml.sax.Attributes
import org.xml.sax.helpers.DefaultHandler
class ThreadLookupAPIResponseHandler : KoinComponent, DefaultHandler() {
private val log by LoggerDelegate()
private val supportedHosts: List<Host> = getKoin().getAll()
private val settingsService: SettingsService by inject()
private var error: String = ""
private val hostMap: MutableMap<Host, Int> = mutableMapOf()
private val postItemList: MutableList<PostItem> = mutableListOf()
private val imageItemList: MutableList<ImageItem> = mutableListOf()
private lateinit var threadItem: ThreadItem
private lateinit var threadId: String
private lateinit var threadTitle: String
private lateinit var forum: String
private lateinit var securityToken: String
private lateinit var postId: String
private lateinit var postTitle: String
private var postCounter: Int = 0
val result: ThreadItem
get() = threadItem
override fun startDocument() {}
override fun startElement(
uri: String, localName: String, qName: String, attributes: Attributes
) {
when (qName.lowercase()) {
"error" -> error = attributes.getValue("details")
"thread" -> {
threadId = attributes.getValue("id")?.trim() ?: ""
threadTitle = attributes.getValue("title")?.trim() ?: ""
}
"forum" -> forum = attributes.getValue("title")?.trim() ?: ""
"user" -> securityToken = attributes.getValue("hash")?.trim() ?: ""
"post" -> {
postId = attributes.getValue("id")?.trim() ?: ""
postCounter = attributes.getValue("number")?.trim()?.toInt() ?: 0
val title = attributes.getValue("title")?.trim() ?: ""
postTitle = title.ifBlank { threadTitle }
}
"image" -> {
val mainLink = attributes.getValue("main_url")?.trim() ?: ""
val thumbLink = attributes.getValue("thumb_url")?.trim() ?: ""
val type = attributes.getValue("type")?.trim() ?: ""
if (type == "linked") {
try {
val host = supportedHosts.first {
it.isSupported(mainLink)
}.let { host ->
hostMap.computeIfAbsent(host) { 0 }
hostMap[host] = hostMap[host]!! + 1
host
}
imageItemList.add(ImageItem(mainLink, thumbLink, host))
} catch (e: Exception) {
log.warn("Unsupported link: $mainLink", e)
}
}
}
}
}
override fun endElement(uri: String, localName: String, qName: String) {
if ("post".equals(qName, true)) {
if (imageItemList.isNotEmpty()) {
postItemList.add(
PostItem(
threadId,
threadTitle,
postId,
postCounter,
postTitle,
imageItemList.size,
"${settingsService.settings.viperSettings.host}/threads/?p=$postId&viewfull=1#post$postId",
hostMap.toMap().map { Pair(it.key.hostId, it.value) },
securityToken,
forum,
imageItemList.toList()
)
)
}
imageItemList.clear()
hostMap.clear()
}
}
override fun endDocument() {
threadItem = ThreadItem(threadId, threadTitle, securityToken, forum, postItemList.toList())
}
}
@@ -0,0 +1,17 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.Image
import java.util.*
interface ImageRepository {
fun save(image: Image): Image
fun save(imageList: List<Image>)
fun deleteAllByPostId(postId: String)
fun findByPostId(postId: String): List<Image>
fun countError(): Int
fun findByPostIdAndIsNotCompleted(postId: String): List<Image>
fun stopByPostIdAndIsNotCompleted(postId: String): Int
fun findByPostIdAndIsError(postId: String): List<Image>
fun findById(id: Long): Optional<Image>
fun update(image: Image)
}
@@ -0,0 +1,13 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.LogEvent
import java.util.*
interface LogEventRepository {
fun save(logEvent: LogEvent): LogEvent
fun update(logEvent: LogEvent): LogEvent
fun findAll(): List<LogEvent>
fun findById(id: Long): Optional<LogEvent>
fun delete(id: Long)
fun deleteAll()
}
@@ -0,0 +1,10 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.Metadata
import java.util.*
interface MetadataRepository {
fun save(metadata: Metadata): Metadata
fun findByPostId(postId: String): Optional<Metadata>
fun deleteByPostId(postId: String): Int
}
@@ -0,0 +1,17 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.Post
import java.util.*
interface PostDownloadStateRepository {
fun save(post: Post): Post
fun findByPostId(postId: String): Optional<Post>
fun findById(id: Long): Optional<Post>
fun findCompleted(): List<String>
fun findAll(): List<Post>
fun existByPostId(postId: String): Boolean
fun setDownloadingToStopped(): Int
fun deleteByPostId(postId: String): Int
fun update(post: Post)
fun update(post: List<Post>)
}
@@ -0,0 +1,13 @@
package me.mnlr.vripper.repositories
import me.mnlr.vripper.entities.Thread
import java.util.*
interface ThreadRepository {
fun save(thread: Thread): Thread
fun findByThreadId(threadId: String): Optional<Thread>
fun findAll(): List<Thread>
fun findById(id: Long): Optional<Thread>
fun deleteByThreadId(threadId: String): Int
fun deleteAll()
}
@@ -0,0 +1,120 @@
package me.mnlr.vripper.repositories.impl
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.repositories.ImageRepository
import me.mnlr.vripper.tables.ImageTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
class ImageRepositoryImpl : ImageRepository {
override fun save(image: Image): Image {
val id = ImageTable.insertAndGetId {
it[current] = image.current
it[host] = image.host
it[index] = image.index
it[postId] = image.postId
it[status] = image.status.name
it[total] = image.total
it[url] = image.url
it[thumbUrl] = image.thumbUrl
it[postIdRef] = image.postIdRef
}.value
return image.copy(id = id)
}
override fun save(imageList: List<Image>) {
ImageTable.batchInsert(imageList, shouldReturnGeneratedValues = false) {
this[ImageTable.current] = it.current
this[ImageTable.host] = it.host
this[ImageTable.index] = it.index
this[ImageTable.postId] = it.postId
this[ImageTable.status] = it.status.name
this[ImageTable.total] = it.total
this[ImageTable.url] = it.url
this[ImageTable.thumbUrl] = it.thumbUrl
this[ImageTable.postIdRef] = it.postIdRef
}
}
override fun deleteAllByPostId(postId: String) {
ImageTable.deleteWhere { ImageTable.postId eq postId }
}
override fun findByPostId(postId: String): List<Image> {
return ImageTable.select {
ImageTable.postId eq postId
}.map(this::transform)
}
override fun countError(): Int {
return ImageTable
.slice(ImageTable.id)
.select { ImageTable.status eq Status.ERROR.name }
.count().toInt()
}
override fun findByPostIdAndIsNotCompleted(postId: String): List<Image> {
return ImageTable
.select {
(ImageTable.postId eq postId) and (ImageTable.status neq Status.FINISHED.name)
}.map(this::transform)
}
override fun stopByPostIdAndIsNotCompleted(postId: String): Int {
return ImageTable.update({ (ImageTable.postId eq postId) and (ImageTable.status neq Status.FINISHED.name) }) {
it[status] = Status.STOPPED.name
}
}
override fun findByPostIdAndIsError(postId: String): List<Image> {
return ImageTable.select {
(ImageTable.postId eq postId) and (ImageTable.status eq Status.ERROR.name)
}.map(this::transform)
}
override fun findById(id: Long): Optional<Image> {
val result = ImageTable.select {
ImageTable.id eq id
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun update(image: Image) {
ImageTable.update({ ImageTable.id eq image.id }) {
it[status] = image.status.name
it[current] = image.current
it[total] = image.total
}
}
private fun transform(resultRow: ResultRow): Image {
val id = resultRow[ImageTable.id].value
val postId = resultRow[ImageTable.postId]
val url = resultRow[ImageTable.url]
val thumbUrl = resultRow[ImageTable.thumbUrl]
val host = resultRow[ImageTable.host]
val index = resultRow[ImageTable.index]
val current = resultRow[ImageTable.current]
val total = resultRow[ImageTable.total]
val status = Status.valueOf(resultRow[ImageTable.status])
val postIdRef = resultRow[ImageTable.postIdRef]
return Image(
id,
postId,
url,
thumbUrl,
host,
index,
postIdRef,
total,
current,
status
)
}
}
@@ -0,0 +1,33 @@
package me.mnlr.vripper.repositories.impl
import me.mnlr.vripper.entities.LogEvent
import me.mnlr.vripper.repositories.LogEventRepository
import java.util.*
class LogEventRepositoryImpl : LogEventRepository {
@Synchronized
override fun save(logEvent: LogEvent): LogEvent {
return logEvent
}
override fun update(logEvent: LogEvent): LogEvent {
return logEvent
}
override fun findById(id: Long): Optional<LogEvent> {
return Optional.empty()
}
override fun findAll(): List<LogEvent> {
return emptyList()
}
override fun delete(id: Long) {
}
override fun deleteAll() {
}
}
@@ -0,0 +1,20 @@
package me.mnlr.vripper.repositories.impl
import me.mnlr.vripper.entities.Metadata
import me.mnlr.vripper.repositories.MetadataRepository
import java.util.*
class MetadataRepositoryImpl: MetadataRepository {
override fun save(metadata: Metadata): Metadata {
return metadata
}
override fun findByPostId(postId: String): Optional<Metadata> {
return Optional.empty()
}
override fun deleteByPostId(postId: String): Int {
return 0
}
}
@@ -0,0 +1,145 @@
package me.mnlr.vripper.repositories.impl
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.tables.PostTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
class PostDownloadStateRepositoryImpl :
PostDownloadStateRepository {
private val delimiter = ";"
override fun save(post: Post): Post {
val id = PostTable.insertAndGetId {
it[done] = post.done
it[hosts] = java.lang.String.join(delimiter, post.hosts)
it[outputPath] = post.downloadDirectory
it[postId] = post.postId
it[status] = post.status.name
it[threadId] = post.threadId
it[postTitle] = post.postTitle
it[threadTitle] = post.threadTitle
it[forum] = post.forum
it[total] = post.total
it[url] = post.url
it[token] = post.token
it[addedAt] = post.addedOn
it[rank] = post.rank
}.value
return post.copy(id = id)
}
override fun findByPostId(postId: String): Optional<Post> {
val result = PostTable.select {
PostTable.postId eq postId
}.map { transform(it) }
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun findCompleted(): List<String> {
return PostTable.slice(PostTable.postId).select {
(PostTable.status eq Status.FINISHED.name) and (PostTable.done greaterEq PostTable.total)
}.map { it[PostTable.postId] }
}
override fun findById(id: Long): Optional<Post> {
val result = PostTable.select {
PostTable.id eq id
}.map { transform(it) }
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun findAll(): List<Post> {
return PostTable.selectAll().map { transform(it) }
}
override fun existByPostId(postId: String): Boolean {
return PostTable.slice(PostTable.id).select { PostTable.postId eq postId }.count() > 0
}
override fun setDownloadingToStopped(): Int {
return PostTable.update({ (PostTable.status eq Status.DOWNLOADING.name) or (PostTable.status eq Status.PENDING.name) }) {
it[status] = Status.STOPPED.name
}
}
override fun deleteByPostId(postId: String): Int {
return PostTable.deleteWhere { PostTable.postId eq postId }
}
override fun update(post: Post) {
PostTable.update({ PostTable.id eq post.id }) {
it[status] = post.status.name
it[done] = post.done
it[rank] = post.rank
}
}
override fun update(post: List<Post>) {
PostTable.batchReplace(post, shouldReturnGeneratedValues = false) {
this[PostTable.id] = it.id
this[PostTable.status] = it.status.name
this[PostTable.done] = it.done
this[PostTable.total] = it.total
this[PostTable.rank] = it.rank
this[PostTable.hosts] = java.lang.String.join(delimiter, it.hosts)
this[PostTable.outputPath] = it.downloadDirectory
this[PostTable.postId] = it.postId
this[PostTable.threadId] = it.threadId
this[PostTable.postTitle] = it.postTitle
this[PostTable.threadTitle] = it.threadTitle
this[PostTable.forum] = it.forum
this[PostTable.url] = it.url
this[PostTable.token] = it.token
}
}
private fun transform(resultRow: ResultRow): Post {
val id = resultRow[PostTable.id].value
val status = Status.valueOf(resultRow[PostTable.status])
val postId = resultRow[PostTable.postId]
val threadId = resultRow[PostTable.threadId]
val postTitle = resultRow[PostTable.postTitle]
val threadTitle = resultRow[PostTable.threadTitle]
val forum = resultRow[PostTable.forum]
val url = resultRow[PostTable.url]
val token = resultRow[PostTable.token]
val done = resultRow[PostTable.done]
val total = resultRow[PostTable.total]
val hosts =
resultRow[PostTable.hosts].split(delimiter).dropLastWhile { it.isEmpty() }.toSet()
val downloadDirectory = resultRow[PostTable.outputPath]
val addedOn = resultRow[PostTable.addedAt]
val rank = resultRow[PostTable.rank]
return Post(
id,
postTitle,
threadTitle,
forum,
url,
token,
postId,
threadId,
total,
hosts,
downloadDirectory,
addedOn,
status,
done,
rank
)
}
}
@@ -0,0 +1,64 @@
package me.mnlr.vripper.repositories.impl
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.repositories.ThreadRepository
import me.mnlr.vripper.tables.ThreadTable
import org.jetbrains.exposed.sql.*
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import java.util.*
class ThreadRepositoryImpl : ThreadRepository {
override fun save(thread: Thread): Thread {
val id = ThreadTable.insertAndGetId {
it[title] = thread.title
it[total] = thread.total
it[url] = thread.link
it[threadId] = thread.threadId
}.value
return thread.copy(id = id)
}
override fun findByThreadId(threadId: String): Optional<Thread> {
val result = ThreadTable.select {
ThreadTable.threadId eq threadId
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun findAll(): List<Thread> {
return ThreadTable.selectAll().map(this::transform)
}
override fun findById(id: Long): Optional<Thread> {
val result = ThreadTable.select {
ThreadTable.id eq id
}.map(this::transform)
return if (result.isEmpty()) {
Optional.empty()
} else {
Optional.of(result.first())
}
}
override fun deleteByThreadId(threadId: String): Int {
return ThreadTable.deleteWhere { ThreadTable.threadId eq threadId }
}
override fun deleteAll() {
ThreadTable.deleteAll()
}
private fun transform(resultRow: ResultRow): Thread {
val id = resultRow[ThreadTable.id].value
val title = resultRow[ThreadTable.title]
val url = resultRow[ThreadTable.url]
val threadId = resultRow[ThreadTable.threadId]
val total = resultRow[ThreadTable.total]
return Thread(id, title, url, threadId, total)
}
}
@@ -0,0 +1,5 @@
package me.mnlr.vripper.services
object AppVersion {
const val VERSION: String = "4.4.0"
}
@@ -0,0 +1,272 @@
package me.mnlr.vripper.services
import com.github.benmanes.caffeine.cache.Caffeine
import com.github.benmanes.caffeine.cache.LoadingCache
import me.mnlr.vripper.entities.Image
import me.mnlr.vripper.entities.Metadata
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.entities.Thread
import me.mnlr.vripper.entities.domain.Status
import me.mnlr.vripper.event.*
import me.mnlr.vripper.model.PostItem
import me.mnlr.vripper.repositories.ImageRepository
import me.mnlr.vripper.repositories.MetadataRepository
import me.mnlr.vripper.repositories.PostDownloadStateRepository
import me.mnlr.vripper.repositories.ThreadRepository
import org.jetbrains.exposed.sql.transactions.transaction
import java.util.*
import kotlin.io.path.pathString
class DataTransaction(
private val settingsService: SettingsService,
private val postDownloadStateRepository: PostDownloadStateRepository,
private val imageRepository: ImageRepository,
private val threadRepository: ThreadRepository,
private val metadataRepository: MetadataRepository,
private val eventBus: EventBus,
) {
private val imageIdCache: LoadingCache<Long, Optional<Image>> = Caffeine.newBuilder().build {
_findImageById(it)
}
private val postIdCache: LoadingCache<Long, Optional<Post>> = Caffeine.newBuilder().build {
_findPostById(it)
}
private fun save(post: Post): Post {
return transaction { postDownloadStateRepository.save(post) }
}
fun update(post: Post) {
transaction { postDownloadStateRepository.update(post) }
postIdCache.invalidate(post.id)
eventBus.publishEvent(PostUpdateEvent(post))
}
fun save(thread: Thread) {
val savedThread = transaction { threadRepository.save(thread) }
eventBus.publishEvent(
ThreadCreateEvent(
savedThread
)
)
}
fun update(image: Image) {
transaction { imageRepository.update(image) }
imageIdCache.invalidate(image.id)
eventBus.publishEvent(ImageUpdateEvent(image))
}
fun exists(postId: String): Boolean {
return transaction { postDownloadStateRepository.existByPostId(postId) }
}
fun newPost(postItem: PostItem): Post {
val (post, images) = transaction {
val post = save(
Post(
postTitle = postItem.title,
url = postItem.url,
token = postItem.securityToken,
postId = postItem.postId,
threadId = postItem.threadId,
total = postItem.imageCount,
hosts = postItem.hosts.map { "${it.first} (${it.second})" }.toSet(),
threadTitle = postItem.threadTitle,
forum = postItem.forum,
downloadDirectory = PathUtils.calculateDownloadPath(
postItem.forum,
postItem.threadTitle,
postItem.title,
postItem.postId,
settingsService.settings
).pathString
)
)
val images = postItem.imageItemList.mapIndexed { index, imageItem ->
Image(
postId = postItem.postId,
url = imageItem.mainLink,
thumbUrl = imageItem.thumbLink,
host = imageItem.host.hostId,
index = index,
postIdRef = post.id!!
)
}
save(images)
sortPostsByRank()
Pair(post, images)
}
eventBus.publishEvent(PostCreateEvent(post))
images.forEach {
eventBus.publishEvent(ImageCreateEvent(it))
}
return post
}
private fun save(images: List<Image>) {
transaction { imageRepository.save(images) }
}
fun finishPost(post: Post) {
val imagesInErrorStatus = findByPostIdAndIsError(post.postId)
if (imagesInErrorStatus.isNotEmpty()) {
post.status = Status.ERROR
update(post)
} else {
if (post.done < post.total) {
post.status = Status.STOPPED
update(post)
} else {
post.status = Status.FINISHED
transaction {
update(post)
if (settingsService.settings.downloadSettings.clearCompleted) {
remove(listOf(post.postId))
}
}
}
}
}
private fun findByPostIdAndIsError(postId: String): List<Image> {
return transaction { imageRepository.findByPostIdAndIsError(postId) }
}
private fun remove(postIds: List<String>) {
transaction {
for (postId in postIds) {
imageRepository.deleteAllByPostId(postId)
metadataRepository.deleteByPostId(postId)
postDownloadStateRepository.deleteByPostId(postId)
}
sortPostsByRank()
}
postIds.forEach {
eventBus.publishEvent(PostDeleteEvent(it))
}
}
fun removeThread(threadId: String) {
transaction { threadRepository.deleteByThreadId(threadId) }
eventBus.publishEvent(ThreadDeleteEvent(threadId))
}
fun clearCompleted(): List<String> {
val completed = transaction { postDownloadStateRepository.findCompleted() }
remove(completed)
return completed
}
fun removeAll(postIds: List<String>?) {
if (postIds != null) {
remove(postIds)
} else {
remove(findAllPosts().map(Post::postId))
}
}
fun stopImagesByPostIdAndIsNotCompleted(postId: String) {
transaction { imageRepository.stopByPostIdAndIsNotCompleted(postId) }
}
@Synchronized
fun setMetadata(post: Post, metadata: Metadata) {
if (metadataRepository.findByPostId(post.postId).isEmpty) {
metadata.postIdRef = post.id
metadataRepository.save(metadata)
}
}
fun clearQueueLinks() {
transaction { threadRepository.deleteAll() }
eventBus.publishEvent(ThreadClearEvent())
}
@Synchronized
fun sortPostsByRank() {
val post =
findAllPosts().sortedWith(Comparator.comparing(Post::addedOn))
for (i in post.indices) {
post[i].rank = i
}
update(post)
}
private fun update(postList: List<Post>) {
transaction { postDownloadStateRepository.update(postList) }
postList.forEach { postIdCache.invalidate(it.id) }
postList.forEach {
eventBus.publishEvent(PostUpdateEvent(it))
}
}
fun setDownloadingToStopped() {
transaction { postDownloadStateRepository.setDownloadingToStopped() }
}
fun findAllPosts(): List<Post> {
return transaction { postDownloadStateRepository.findAll() }
}
fun findPostById(id: Long): Optional<Post> {
return postIdCache[id]
}
private fun _findPostById(id: Long): Optional<Post> {
return transaction { postDownloadStateRepository.findById(id) }
}
fun findImagesByPostId(postId: String): List<Image> {
return transaction { imageRepository.findByPostId(postId) }
}
fun findImageById(id: Long): Optional<Image> {
return imageIdCache[id]
}
private fun _findImageById(id: Long): Optional<Image> {
return transaction { imageRepository.findById(id) }
}
fun findAllThreads(): List<Thread> {
return transaction { threadRepository.findAll() }
}
fun findThreadById(id: Long): Optional<Thread> {
return transaction {
threadRepository.findById(id)
}
}
fun findByPostIdAndIsNotCompleted(postId: String): List<Image> {
return transaction { imageRepository.findByPostIdAndIsNotCompleted(postId) }
}
fun countImagesInError(): Int {
return transaction { imageRepository.countError() }
}
fun findPostsByPostId(postId: String): Optional<Post> {
return transaction { postDownloadStateRepository.findByPostId(postId) }
}
fun findThreadByThreadId(threadId: String): Optional<Thread> {
return transaction { threadRepository.findByThreadId(threadId) }
}
}
@@ -0,0 +1,27 @@
package me.mnlr.vripper.services
import liquibase.Contexts
import liquibase.LabelExpression
import liquibase.Liquibase
import liquibase.database.Database
import liquibase.database.DatabaseFactory
import liquibase.database.jvm.JdbcConnection
import liquibase.resource.ClassLoaderResourceAccessor
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import java.sql.DriverManager
object DatabaseMigration {
fun update() {
val database: Database = DatabaseFactory.getInstance()
.findCorrectDatabaseImplementation(JdbcConnection(DriverManager.getConnection("jdbc:h2:file:$baseDir/$BASE_DIR_NAME/vripper;DB_CLOSE_DELAY=-1;")))
Liquibase(
"db.changelog-master.xml",
ClassLoaderResourceAccessor(),
database
).use { liquibase ->
liquibase.update(Contexts(), LabelExpression())
}
}
}
@@ -0,0 +1,53 @@
package me.mnlr.vripper.services
import kotlinx.coroutines.*
import me.mnlr.vripper.event.DownloadSpeedEvent
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.QueueStateEvent
import me.mnlr.vripper.formatSI
import me.mnlr.vripper.model.DownloadSpeed
import java.util.concurrent.atomic.AtomicLong
class DownloadSpeedService(
private val eventBus: EventBus,
) {
companion object {
const val DOWNLOAD_POLL_RATE = 2500
}
private val coroutineScope = CoroutineScope(Dispatchers.Default)
private val bytesCount = AtomicLong(0)
private var job: Job? = null
fun init() {
coroutineScope.launch {
eventBus.subscribe<QueueStateEvent> {
if (it.queueState.running + it.queueState.remaining > 0) {
if (job == null || job?.isActive == false) {
job = coroutineScope.launch {
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(0L.formatSI())))
while (isActive) {
delay(DOWNLOAD_POLL_RATE.toLong())
val newValue = bytesCount.getAndSet(0)
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(((newValue * 1000) / DOWNLOAD_POLL_RATE).formatSI())))
}
}
}
} else {
job?.cancel()
coroutineScope.launch {
delay(DOWNLOAD_POLL_RATE + 500L)
eventBus.publishEvent(DownloadSpeedEvent(DownloadSpeed(0L.formatSI())))
}
}
}
}
}
fun reportDownloadedBytes(count: Long) {
bytesCount.addAndGet(count)
}
}
@@ -0,0 +1,150 @@
package me.mnlr.vripper.services
import kotlinx.coroutines.*
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import org.apache.hc.client5.http.async.methods.SimpleRequestBuilder
import org.apache.hc.client5.http.classic.methods.HttpGet
import org.apache.hc.client5.http.classic.methods.HttpHead
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.classic.methods.HttpUriRequest
import org.apache.hc.client5.http.config.ConnectionConfig
import org.apache.hc.client5.http.config.RequestConfig
import org.apache.hc.client5.http.cookie.StandardCookieSpec
import org.apache.hc.client5.http.impl.DefaultRedirectStrategy
import org.apache.hc.client5.http.impl.async.HttpAsyncClientBuilder
import org.apache.hc.client5.http.impl.async.HttpAsyncClients
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManagerBuilder
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.pool.PoolConcurrencyPolicy
import org.apache.hc.core5.pool.PoolReusePolicy
import org.apache.hc.core5.util.TimeValue
import org.apache.hc.core5.util.Timeout
import java.net.URI
class HTTPService(
val eventBus: EventBus,
settingsService: SettingsService
) {
companion object {
const val USER_AGENT =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0"
}
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
private lateinit var pcm: PoolingAsyncClientConnectionManager
private lateinit var rc: RequestConfig
private lateinit var cc: ConnectionConfig
lateinit var clientBuilder: HttpAsyncClientBuilder
private var connectionTimeout: Long = settingsService.settings.connectionSettings.timeout
fun init() {
buildRequestConfig()
buildConnectionConfig()
buildConnectionPool()
buildClientBuilder()
coroutineScope.launch {
pcm.closeIdle(TimeValue.ofSeconds(60))
delay(15000)
}
coroutineScope.launch {
eventBus
.subscribe<SettingsUpdateEvent> {
if (connectionTimeout != it.settings.connectionSettings.timeout) {
connectionTimeout = it.settings.connectionSettings.timeout
buildRequestConfig()
buildConnectionConfig()
pcm.close()
buildConnectionPool()
buildClientBuilder()
}
}
}
}
private fun buildConnectionPool() {
pcm = PoolingAsyncClientConnectionManagerBuilder.create()
.setPoolConcurrencyPolicy(PoolConcurrencyPolicy.STRICT)
.setConnPoolPolicy(PoolReusePolicy.LIFO)
.setDefaultConnectionConfig(cc)
.setMaxConnTotal(Int.MAX_VALUE)
.setMaxConnPerRoute(Int.MAX_VALUE)
.build()
}
private fun buildRequestConfig() {
rc = RequestConfig.custom()
.setConnectionRequestTimeout(Timeout.ofSeconds(connectionTimeout))
.setCookieSpec(StandardCookieSpec.RELAXED)
.build()
}
private fun buildConnectionConfig() {
cc = ConnectionConfig.custom()
.setConnectTimeout(Timeout.ofSeconds(connectionTimeout))
.setSocketTimeout(Timeout.ofSeconds(connectionTimeout))
.setTimeToLive(TimeValue.ofMinutes(10))
.build()
}
private fun buildClientBuilder() {
clientBuilder = HttpAsyncClients.custom()
.setConnectionManager(pcm)
.setRedirectStrategy(DefaultRedirectStrategy.INSTANCE)
.disableAutomaticRetries()
.setDefaultRequestConfig(rc)
}
fun buildHttpGet(url: String, context: HttpClientContext): HttpGet {
val httpGet = HttpGet(url.replace(" ", "+"))
httpGet.addHeader("User-Agent", USER_AGENT)
addToContext(context, httpGet)
return httpGet
}
fun buildHttpHead(url: String, context: HttpClientContext): HttpHead {
SimpleRequestBuilder.head(url)
val httpHead = HttpHead(url.replace(" ", "+"))
httpHead.addHeader("User-Agent", USER_AGENT)
addToContext(context, httpHead)
return httpHead
}
fun buildHttpPost(url: String, context: HttpClientContext): HttpPost {
val httpPost = HttpPost(url.replace(" ", "+"))
httpPost.addHeader("User-Agent", USER_AGENT)
addToContext(context, httpPost)
return httpPost
}
fun buildHttpGet(uri: URI, context: HttpClientContext): HttpGet {
val httpGet = HttpGet(uri)
httpGet.addHeader("User-Agent", USER_AGENT)
addToContext(context, httpGet)
return httpGet
}
fun addToContext(context: HttpClientContext, request: HttpUriRequest) {
val contextAttributes =
context.getAttribute(
ContextAttributes.CONTEXT_ATTRIBUTES,
ContextAttributes::class.java
)
if (contextAttributes != null) {
synchronized(contextAttributes.requests) {
contextAttributes.requests.add(request)
}
}
}
class ContextAttributes {
val requests: MutableList<HttpUriRequest> = mutableListOf()
companion object {
const val CONTEXT_ATTRIBUTES = "CONTEXT_ATTRIBUTES"
}
}
}
@@ -0,0 +1,22 @@
package me.mnlr.vripper.services
import me.mnlr.vripper.exception.HtmlProcessorException
import org.htmlcleaner.CleanerProperties
import org.htmlcleaner.DomSerializer
import org.htmlcleaner.HtmlCleaner
import org.w3c.dom.Document
import java.io.InputStream
object HtmlProcessorService {
@Throws(HtmlProcessorException::class)
fun clean(htmlContent: InputStream): Document {
return try {
htmlContent.use {
val clean = HtmlCleaner().clean(htmlContent)
DomSerializer(CleanerProperties()).createDOM(clean)
}
} catch (e: Exception) {
throw HtmlProcessorException(e)
}
}
}
@@ -0,0 +1,34 @@
package me.mnlr.vripper.services
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.model.Settings
import java.nio.file.Path
object PathUtils {
private val log by LoggerDelegate()
fun calculateDownloadPath(forum: String, threadTitle: String, postTitle: String, postId: String, settings: Settings): Path {
var downloadDirectory =
if (settings.downloadSettings.forumSubfolder) Path.of(settings.downloadSettings.downloadPath, sanitize(forum)) else Path.of(
settings.downloadSettings.downloadPath
)
downloadDirectory = if (settings.downloadSettings.threadSubLocation) downloadDirectory.resolve(threadTitle) else downloadDirectory
downloadDirectory = downloadDirectory.resolve(if (settings.downloadSettings.appendPostId) "${sanitize(postTitle)}_${postId}" else sanitize(
postTitle
))
return downloadDirectory
}
/**
* Will sanitize the image name and remove extension
*
* @param path
* @return Sanitized local path string
*/
fun sanitize(path: String): String {
val sanitizedPath =
path.replace("\\.|\\\\|/|\\||:|\\?|\\*|\"|<|>|\\p{Cntrl}".toRegex(), "_")
log.debug(String.format("%s sanitized to %s", path, sanitizedPath))
return sanitizedPath
}
}
@@ -0,0 +1,49 @@
package me.mnlr.vripper.services
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import net.jodah.failsafe.RetryPolicy
import net.jodah.failsafe.event.ExecutionAttemptedEvent
import java.time.temporal.ChronoUnit
class RetryPolicyService(
val eventBus: EventBus,
settingsService: SettingsService
) {
private val log by LoggerDelegate()
private var maxAttempts: Int = settingsService.settings.connectionSettings.maxAttempts
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
fun init() {
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
if (maxAttempts != it.settings.connectionSettings.maxAttempts) {
maxAttempts = it.settings.connectionSettings.maxAttempts
}
}
}
}
fun <T> buildRetryPolicyForDownload(): RetryPolicy<T> {
return RetryPolicy<T>()
.withDelay(2, 5, ChronoUnit.SECONDS)
.withMaxAttempts(maxAttempts)
.onFailedAttempt {
log.warn("#${it.attemptCount} tries failed", it.lastFailure)
}
}
fun <T> buildGenericRetryPolicy(): RetryPolicy<T> {
return RetryPolicy<T>()
.withDelay(2, 5, ChronoUnit.SECONDS)
.withMaxAttempts(maxAttempts)
.onFailedAttempt { e: ExecutionAttemptedEvent<T> ->
log.warn("#${e.attemptCount} tries failed", e.lastFailure)
}
}
}
@@ -0,0 +1,197 @@
package me.mnlr.vripper.services
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.registerKotlinModule
import me.mnlr.vripper.ApplicationProperties.BASE_DIR_NAME
import me.mnlr.vripper.ApplicationProperties.baseDir
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.exception.ValidationException
import me.mnlr.vripper.model.Settings
import org.apache.commons.codec.digest.DigestUtils
import java.io.FileWriter
import java.nio.file.*
class SettingsService(private val eventBus: EventBus) {
private val log by LoggerDelegate()
private val configPath = Paths.get(baseDir, BASE_DIR_NAME, "config.yml")
private val customProxiesPath = Paths.get(baseDir, BASE_DIR_NAME, "proxies.json")
private val om = ObjectMapper(YAMLFactory())
private val proxies: MutableSet<String> = HashSet()
var settings = Settings()
init {
om.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.findAndRegisterModules().registerKotlinModule()
init()
}
private fun init() {
loadViperProxies()
restore()
eventBus.publishEvent(SettingsUpdateEvent(settings))
}
private fun loadViperProxies() {
try {
SettingsService::class.java.getResourceAsStream("proxies.json")?.use {
val defaultProxies: List<String> = om.readValue(it)
val customProxies: List<String> = if (customProxiesPath.toFile()
.exists() && Files.isRegularFile(customProxiesPath)
) {
try {
om.readValue(
customProxiesPath.toFile()
)
} catch (e: Exception) {
emptyList()
}
} else {
try {
FileWriter(customProxiesPath.toFile()).use { fw ->
fw.append("[]").append(System.lineSeparator())
fw.flush()
}
} catch (e: Exception) {
log.warn(
"Unable to create " + customProxiesPath.toFile().absolutePath, e
)
}
emptyList()
}
proxies.addAll(defaultProxies)
proxies.addAll(customProxies)
}
} catch (e: Exception) {
log.error("Failed to load vipergirls proxies list", e)
}
proxies.add("https://vipergirls.to")
}
fun getProxies(): List<String> {
return ArrayList(proxies)
}
fun newSettings(settings: Settings) {
if (settings.viperSettings.login) {
if (this.settings.viperSettings.password != settings.viperSettings.password) {
settings.viperSettings.password =
DigestUtils.md5Hex(settings.viperSettings.password)
}
} else {
settings.viperSettings.username = ""
settings.viperSettings.password = ""
settings.viperSettings.thanks = false
settings.viperSettings.login = false
}
check(settings)
this.settings = settings
save()
eventBus.publishEvent(SettingsUpdateEvent(settings))
}
private fun restore() {
try {
if (configPath.toFile().exists()) {
settings = om.readValue(configPath.toFile())
}
} catch (e: Exception) {
log.error("Failed restore user settings", e)
settings = Settings()
}
if (!proxies.contains(settings.viperSettings.host)) {
settings.viperSettings.host = "https://vipergirls.to"
}
try {
check(settings)
} catch (e: ValidationException) {
log.error("Your settings are invalid, using default settings", e)
settings = Settings()
}
save()
}
fun save() {
try {
Files.write(
configPath,
om.writeValueAsBytes(settings),
StandardOpenOption.CREATE,
StandardOpenOption.WRITE,
StandardOpenOption.TRUNCATE_EXISTING,
StandardOpenOption.SYNC
)
} catch (e: Exception) {
log.error("Failed to store user settings", e)
}
}
@Throws(ValidationException::class)
fun check(settings: Settings) {
val path: Path = try {
Paths.get(settings.downloadSettings.downloadPath)
} catch (e: InvalidPathException) {
throw ValidationException(
String.format(
"%s is invalid", settings.downloadSettings.downloadPath
)
)
}
if (!Files.exists(path)) {
throw ValidationException(
String.format(
"%s does not exist", settings.downloadSettings.downloadPath
)
)
} else if (!Files.isDirectory(path)) {
throw ValidationException(
String.format(
"%s is not a directory", settings.downloadSettings.downloadPath
)
)
}
if (settings.downloadSettings.autoQueueThreshold < 0) {
throw ValidationException("Invalid auto queue settings, value must be a positive integer")
}
if (settings.connectionSettings.maxTotalThreads < 0 || settings.connectionSettings.maxTotalThreads > 12) {
throw ValidationException(
"Invalid max global concurrent download settings, values must be in [0,12]"
)
}
if (settings.connectionSettings.maxThreads < 1 || settings.connectionSettings.maxThreads > 4) {
throw ValidationException("Invalid max concurrent download settings, values must be in [1,4]")
}
if (settings.connectionSettings.timeout < 1 || settings.connectionSettings.timeout > 300) {
throw ValidationException(
"Invalid connection timeout settings, values must be in [1,300]"
)
}
if (settings.connectionSettings.maxAttempts < 1 || settings.connectionSettings.maxAttempts > 10) {
throw ValidationException("Invalid maximum attempts settings, values must be in [1,10]")
}
if (settings.maxEventLog < 100 || settings.maxEventLog > 10000) {
throw ValidationException(
"Invalid maximum event log record settings, values must be in [100,10000]"
)
}
if (settings.clipboardSettings.pollingRate < 500) {
throw ValidationException(
"Invalid clipboard monitoring polling rate settings, values must be >= 500"
)
}
}
}
@@ -0,0 +1,38 @@
package me.mnlr.vripper.services
import com.github.benmanes.caffeine.cache.Caffeine
import com.github.benmanes.caffeine.cache.LoadingCache
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.model.ThreadItem
import me.mnlr.vripper.parser.ThreadLookupAPIParser
import java.util.*
import java.util.concurrent.ExecutionException
import java.util.concurrent.TimeUnit
class ThreadCacheService(val eventBus: EventBus) {
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
fun init() {
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
cache.invalidateAll()
}
}
}
private val cache: LoadingCache<String, ThreadItem> =
Caffeine.newBuilder().expireAfterWrite(5, TimeUnit.MINUTES).build { threadId: String ->
ThreadLookupAPIParser(threadId).parse()
}
@Throws(ExecutionException::class)
operator fun get(threadId: String): ThreadItem {
return cache[threadId] ?: throw NoSuchElementException("$threadId does not exist")
}
}
@@ -0,0 +1,151 @@
package me.mnlr.vripper.services
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import me.mnlr.vripper.delegate.LoggerDelegate
import me.mnlr.vripper.entities.Post
import me.mnlr.vripper.event.EventBus
import me.mnlr.vripper.event.SettingsUpdateEvent
import me.mnlr.vripper.event.VGUserLoginEvent
import me.mnlr.vripper.exception.VripperException
import me.mnlr.vripper.model.Settings
import me.mnlr.vripper.tasks.LeaveThanksRunnable
import org.apache.hc.client5.http.cookie.BasicCookieStore
import org.apache.hc.client5.http.cookie.Cookie
import org.apache.hc.client5.http.entity.UrlEncodedFormEntity
import org.apache.hc.client5.http.protocol.HttpClientContext
import org.apache.hc.core5.http.NameValuePair
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.message.BasicNameValuePair
import java.util.concurrent.CompletableFuture
class VGAuthService(
private val cm: HTTPService,
private val settingsService: SettingsService,
private val eventBus: EventBus
) {
private val log by LoggerDelegate()
private val coroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
val context: HttpClientContext = HttpClientContext.create()
var loggedUser = ""
private var authenticated = false
fun init() {
context.cookieStore = BasicCookieStore()
coroutineScope.launch {
eventBus.subscribe<SettingsUpdateEvent> {
authenticate(it.settings)
}
}
authenticate(settingsService.settings)
}
private fun authenticate(settings: Settings) {
authenticated = false
if (!settings.viperSettings.login) {
log.debug("Authentication option is disabled")
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
return
}
val username = settings.viperSettings.username
val password = settings.viperSettings.password
if (username.isEmpty() || password.isEmpty()) {
log.error("Cannot authenticate with ViperGirls credentials, username or password is empty")
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
return
}
val postAuth =
cm.buildHttpPost(
settings.viperSettings.host + "/login.php?do=login",
context
)
val params: MutableList<NameValuePair> = ArrayList()
params.add(BasicNameValuePair("vb_login_username", username))
params.add(BasicNameValuePair("cookieuser", "1"))
params.add(BasicNameValuePair("do", "login"))
params.add(BasicNameValuePair("vb_login_md5password", password))
try {
postAuth.entity = UrlEncodedFormEntity(params)
} catch (e: Exception) {
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
log.error(
"Failed to authenticate user with " + settings.viperSettings.host,
e
)
return
}
postAuth.addHeader("Referer", settings.viperSettings.host)
postAuth.addHeader(
"Host",
settings.viperSettings.host.replace("https://", "")
.replace("http://", "")
)
val client = cm.clientBuilder.build()
try {
client.execute(postAuth, context).use { response ->
if (response.code / 100 != 2) {
throw VripperException(
String.format(
"Unexpected response code returned %s", response.code
)
)
}
val responseBody = EntityUtils.toString(response.entity)
log.debug(
String.format(
"Authentication with ViperGirls response body:%n%s",
responseBody
)
)
EntityUtils.consumeQuietly(response.entity)
if (context.cookieStore.cookies.stream()
.map { obj: Cookie -> obj.name }
.noneMatch { e: String -> e == "vg_userid" }
) {
log.error(
String.format(
"Failed to authenticate user with %s, missing vg_userid cookie",
settings.viperSettings.host
)
)
return
}
}
} catch (e: Exception) {
context.cookieStore.clear()
loggedUser = ""
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
log.error(
"Failed to authenticate user with " + settings.viperSettings.host,
e
)
return
}
authenticated = true
loggedUser = username
log.info(String.format("Authenticated: %s", username))
eventBus.publishEvent(VGUserLoginEvent(loggedUser))
}
fun leaveThanks(post: Post) {
CompletableFuture.runAsync(LeaveThanksRunnable(post, authenticated, context))
}
}
@@ -0,0 +1,34 @@
package me.mnlr.vripper.services
import org.w3c.dom.Node
import org.w3c.dom.NodeList
import me.mnlr.vripper.exception.XpathException
import javax.xml.xpath.XPathConstants
import javax.xml.xpath.XPathFactory
object XpathService {
private val xpath = XPathFactory.newInstance().newXPath()
@Throws(XpathException::class)
fun getAsNode(source: Node?, xpathExpression: String?): Node? {
return try {
val value = xpath.compile(xpathExpression).evaluate(source, XPathConstants.NODE)
if(value != null) {
value as Node
} else {
null
}
} catch (e: Exception) {
throw XpathException(e)
}
}
@Throws(XpathException::class)
fun getAsNodeList(source: Node?, xpathExpression: String?): NodeList {
return try {
xpath.compile(xpathExpression).evaluate(source, XPathConstants.NODESET) as NodeList
} catch (e: Exception) {
throw XpathException(e)
}
}
}
@@ -0,0 +1,15 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
object ImageTable : LongIdTable(name = "IMAGE", columnName = "ID") {
val current = long("CURRENT")
val host = varchar("HOST", 255)
val index = integer("INDEX")
val postId = varchar("POST_ID", 255)
val status = varchar("STATUS", 255)
val total = long("TOTAL")
val url = varchar("URL", 3000)
val thumbUrl = varchar("THUMB_URL", 3000)
val postIdRef = long("POST_ID_REF").references(PostTable.id, fkName = "IMAGE_POST_ID_REF_POST_ID_FK")
}
@@ -0,0 +1,22 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
import org.jetbrains.exposed.sql.javatime.datetime
import java.time.LocalDateTime
object PostTable : LongIdTable(name = "POST", columnName = "ID") {
val done = integer("DONE")
val hosts = varchar("HOSTS", 500)
val outputPath = varchar("OUTPUT_PATH", 500)
val postId = varchar("POST_ID", 255)
val status = varchar("STATUS", 255)
val threadId = varchar("THREAD_ID", 255)
val postTitle = varchar("POST_TITLE", 500)
val threadTitle = varchar("THREAD_TITLE", 500)
val forum = varchar("FORUM", 500)
val total = integer("TOTAL")
val url = varchar("URL", 3000)
val token = varchar("TOKEN", 500)
val addedAt = datetime("ADDED_AT").default(LocalDateTime.now())
val rank = integer("RANK").default(0)
}
@@ -0,0 +1,10 @@
package me.mnlr.vripper.tables
import org.jetbrains.exposed.dao.id.LongIdTable
object ThreadTable : LongIdTable("THREAD", columnName = "ID") {
val total = integer("TOTAL").default(0)
val url = varchar("URL", 3000)
val threadId = varchar("THREAD_ID", 255)
val title = varchar("TITLE", 500)
}

Some files were not shown because too many files have changed in this diff Show More