Compare commits

..
1 Commits
Author SHA1 Message Date
death-claw a8755e74e7 v1.4.6
* Fixed a bug when creating image file names
* Remove unnecessary logs
* Fix data.json init
2019-08-06 16:11:10 +01:00
309 changed files with 19753 additions and 24104 deletions
+1 -3
View File
@@ -2,6 +2,4 @@
/**/*/node_modules
/**/*/target
/**/*/build-dir
/**/*/java-runtime
.idea
.vripper
.idea
+2 -685
View File
@@ -1,754 +1,71 @@
# Changelog
## [3.5.4] - 2021-05-29
### Changed
- Fix download queue bug
## [3.5.3] - 2021-05-24
### Changed
- Update donation infos
## [3.5.1] - 2021-05-22
### Changed
- Bug fix
## [3.5.0] - 2021-05-21
### Changed
- Download queue bug fix
- Save window size and position
- Resize confirmation box
- Enhance metadata interruption
- Add Added on and Order columns
## [3.4.2] - 2021-05-10
### Changed
- Fix potential issue with imagebam host
- Enhance download queue logic
- Code source enhancements
- Fix app icons
- Fix race condition bug
## [3.3.8] - 2021-05-05
### Changed
- Fix imagebam host
- Fix font path
## [3.3.6] - 2021-03-20
### Changed
- Optimize build size
## [3.3.5] - 2021-03-18
### Changed
- Bug fixe
## [3.3.4] - 2021-03-18
### Changed
- Bug fixes
## [3.3.3] - 2021-03-13
### Changed
- Remove unnecessary event logs
## [3.3.2] - 2021-03-11
### Changed
- Fix a bug with event logs
- Change event names and add some icons
## [3.3.1] - 2021-02-18
### Added
- Add about menu
### Changed
- Fix README.md
## [3.3.0] - 2021-02-16
### Changed
- Add support for proxies
- Add a Event Logs
- Fix bugs with the database
## [3.2.3] - 2021-01-13
### Changed
- Finished posts are not cleared from the UI
## [3.2.2] - 2021-01-13
### Changed
- Set total download to max value
## [3.2.1] - 2021-01-12
### Added
- Add settings for connection timeout
- Add settings for maximum attempts
### Changed
- Enhance posts rename logic
- Dependencies update
- Switch reactive from rx-java to reactor
## [3.2.0] - 2020-11-16
### Changed
- Fix build issue
## [3.1.0] - 2020-11-16
### Changed
- UI revamp
- Fix bugs
## [3.0.11] - 2020-10-31
### Added
- Support for vipr.im
## [3.0.10] - 2020-10-04
### Changed
- Fix bug on settings update
## [3.0.9] - 2020-09-27
### Changed
- Revert to appending post id to download folders
- Fix a bug that lead to adding duplicate posts
- Remove old unused settings
## [3.0.8] - 2020-08-22
### Changed
- Bug fix which breaks concurrent downloads
## [3.0.7] - 2020-08-22
### Changed
- Fix MacOs build
## [3.0.6] - 2020-08-18
### Changed
- Fix TurboImageHost
## [3.0.5] - 2020-08-17
### Changed
- Update packages
- Host class refactoring
## [3.0.4] - 2020-08-16
### Changed
- Improve rename UI
## [3.0.3] - 2020-08-16
### Changed
- Fix bugs with electron app
## [3.0.2] - 2020-08-15
### Changed
- Fix bugs with download queue
## [3.0.1] - 2020-08-03
### Changed
- Fix bugs with download queue
## [3.0.0] - 2020-08-03
### Changed
- Major rewrites
- Introducing a proper database for persistence
## [2.12.1] - 2020-05-22
### Changed
- Fix alternative name bug
## [2.12.0] - 2020-05-21
### Changed
- Use new user hash from the api
## [2.11.7] - 2020-05-08
### Changed
- Fix bug with paths creation
- Move Open Link button
- Rename rename all title
## [2.11.6] - 2020-05-04
### Changed
- PNG filenames are now correctly named
- Fix issue with special characters in post titles
## [2.11.5] - 2020-05-03
### Changed
- Reduce rows buffer size
## [2.11.4] - 2020-05-03
### Changed
- Code cleanup
- Dependencies update
## [2.11.3] - 2020-04-30
### Changed
- Source code restructure
- UI fixes and enhancements
### Added
- Add menu entry to rename a gallery to first alternative title
- Add menu on toolbar for massive rename to first alternative title
## [2.11.2] - 2020-04-26
### Changed
- Bug fixes
## [2.11.1] - 2020-04-25
### Changed
- Code source maintenance
### Added
- Added option to rename gallery
## [2.11.0] - 2020-03-29
### Changed
- Upgrade Electron to v8
- Upgrade Angular to v9
- App settings are now portable in a json file
- General code source maintenance
### Added
- Added Support for pixroute.com
- Added grab post title from thread option
## [2.10.8] - 2020-01-25
### Changed
- Fix app url
## [2.10.7] - 2020-01-25
### Changed
- Drop SockJs and switch to WebSocket
- Autoreconnect front to back
## [2.10.6] - 2020-01-19
### Changed
- Limit cache size
## [2.10.5] - 2020-01-18
### Changed
- Bug fixes
## [2.10.4] - 2020-01-18
### Changed
- Zero value for total max download will disable the limit
- Add system notification
- Bug fixes
## [2.10.3] - 2020-01-16
### Changed
- Some bug fixes
- Enhance download queue logic
- Clear cache button
- Fix the partial status
- Sanitize URLs with spaces
### Added
- Support for Postimg host
- Support for Imagevenue host
## [2.10.2] - 2019-12-29
### Changed
- Upgrade electron to v7
- Bug fix
## [2.10.1] - 2019-12-27
### Changed
- Bug fix
## [2.10.0] - 2019-12-27
### Changed
- Fix spring dependency bug
### Added
- Add support for imgspice
## [2.9.0] - 2019-12-27
### Changed
- Fix build issue
## [2.9.0] - 2019-12-27
### Changed
- Decrease timeout for fast fail
### Added
- Allow many hosts to run at the same time
- Add title to gallery
- Add global concurrent downloads
- Add counter badge in link collector
## [2.8.1] - 2019-12-24
### Changed
- Fix issue with path creation
## [2.8.0] - 2019-12-24
### Added
- Photo gallery feature
## [2.7.1] - 2019-12-21
### Added
- Add version to title bar
## [2.7.0] - 2019-12-21
### Changed
- Add missing icon
### Added
- Add support for pixxxels
## [2.6.0] - 2019-12-21
### Changed
- Disable download button when nothing selected in Link collector grab menu
- Better quality app icon
- Implement context menu on download tab
- Add stop remove and start button to context menu
- Link will not be removed automatically from link collector tab
- Some UI fixes
## [2.5.1] - 2019-12-09
### Changed
- Disable download button when nothing selected in Link collector grab menu
### Added
- Add hosts column in Link collector grab menu
## [2.5.0] - 2019-12-08
### Changed
- Change loading screen
- UI Fixes
- Fix bug with previews
### Added
- Add link collector feature
- Add support for multiple links at a time
## [2.4.1] - 2019-11-26
### Changed
- Increase attempts for establishing ws connection between front and back end
- Change default download folder to Home folder (installer), App folder (portable)
## [2.4.0] - 2019-11-23
### Changed
- Pimpandhost Host support
## [2.3.2] - 2019-11-23
### Changed
- Fix a bug with the scan component
## [2.3.1] - 2019-11-22
### Changed
- Fix connexion reset issue
- Fix Post Status when autostart option is selected
- Add stop all button
## [2.3.0] - 2019-11-21
### Changed
- UI enhancements and overhaul
- Stability fixes with HTTP connections
- MacOS official support
- Use local sockjs and material icons font
## [2.2.5] - 2019-11-09
### Changed
- Exclude thumbs download for imagezilla and pixhost
- Change retry on fail logic (may help avoid having download errors)
## [2.2.4] - 2019-10-05
### Added
- AppImage target for linux
## [2.2.3] - 2019-10-01
### Changed
- Strip libjvm.so for debian
## [2.2.2] - 2019-10-01
### Changed
- Update product name
## [2.2.1] - 2019-10-01
### Changed
- Update dependency list for deb package
## [2.2.0] - 2019-10-01
### Added
- Embedd JRE within the app
### Changed
- Fix 'Start' on pending items
- Upgrade to Java 11
## [2.1.0] - 2019-09-30
### Added
- Add a create a subfolder per thread option
## [2.0.4] - 2019-09-27
### Changed
- Fix some issues with ImxHost
## [2.0.3] - 2019-09-23
### Changed
- Update deb dependencies
## [2.0.2] - 2019-09-23
### Changed
- Add portable mode
## [2.0.1] - 2019-09-22
### Changed
- Change app name
## [2.0.0] - 2019-09-22
### Added
- Add sub folder option
- Add clear finished option
### Changed
- UI enhancements
- Bug fixes
- Change data and log paths
## [1.6.5] - 2019-09-18
### Changed
- electron update
## [1.6.4] - 2019-09-18
### Changed
- Change menu item name
## [1.6.3] - 2019-09-18
### Changed
- Better multithread management
- Fix partial state UI
## [1.6.2] - 2019-09-17
### Changed
- Rework UI colors
- Fix scan input bug
## [1.6.1] - 2019-09-16
### Changed
- Rework UI
- Fix clipboard monitoring
## [1.6.0] - 2019-09-15
### Changed
- Rework the UI
- Add a dark theme
- Fix the force ordering bug
- Other bug fixes
## [1.5.2] - 2019-09-01
### Added
- imx.to host fix
## [1.5.1] - 2019-08-24
### Added
- Bug fixes
## [1.5.0] - 2019-08-24
### Added
- Display logged in user
- Add post details when clicking on main list items (Status, Post URL, Host)
- Add open download location on main list items menu
- Add support for multi-post threads (Show a menu to select posts to download)
- Add link to image from details list
## [1.4.10] - 2019-08-15
### Changed
- Fix a rename issue on windows
## [1.4.9] - 2019-08-13
### Changed
- Update Built-By to be generic
## [1.4.8] - 2019-08-13
### Changed
- Add option to force image ordering
## [1.4.7] - 2019-08-13
### Changed
- Fixed a bug in image filename extensions
## [1.4.6] - 2019-08-06
### Changed
- Fixed a bug when creating image file names
- Remove unnecessary logs
- Fix data.json init
- Fix data.json init
## [1.4.5] - 2019-08-03
### Changed
- Add min width and height for electron app
### Added
- Add download speed to status bar
## [1.4.4] - 2019-08-01
### Changed
- Fix names for ImgboxHost
- Enhance UI
- Minor fixes
### Added
- Add start and stop button
## [1.4.3] - 2019-07-14
### Changed
- Update some messages
- Fix scrolling issue
## [1.4.2] - 2019-07-14
### Changed
- Fix layout issue in web mode
## [1.4.1] - 2019-07-14
### Changed
- Change title's font
### Added
- Add borders
## [1.4.0] - 2019-07-14
### Changed
- Better error handling on startup
- Better support for linux AppImage
### Added
- Clear all completed
- Add clipboard support
- Add option to remove all posts
- Add confirmation on remove
## [1.3.0-rc0] - 2019-07-04
### Changed
- Change default setting for autostart false -> true
- Change default setting for max concurrent downloads 1 -> 4
### Added
- Desktop app built with electron
- Add the option to remove a post
## [1.2.0] - 2019-06-23
### Added
- Add a shutdown button to gracefully shutdown the app
- Open and redirect the browser to the app address on startup
- Add the option to run the app in headless mode (-Djava.awt.headless=true)
## [1.1.1] - 2019-06-22
### Changed
- Fix a stability issue with progress updates when the app is runninh for a long time
## [1.0.0] - 2019-06-22
### Added
- Add real time communication for progress updates via websocket (SockJS)
- Add settings for download path, max concurrent download, enable/disable integration with vipergirls
- Add integration with vipergitls.to (possibility to auto leave thanks after grab post)
- Add support for acidimg.cc, imagezilla.net, imgbox.com, imx.to, pixhost.to, turboimagehost.com
- Add support for acidimg.cc, imagezilla.net, imgbox.com, imx.to, pixhost.to, turboimagehost.com
+12 -21
View File
@@ -1,35 +1,26 @@
# VRipper!
# Viper Ripper!
This is my spin for a cross platform gallery ripper app for [vipergirls](https://vipergirls.to) website. The purpose of
this project is to build a robust and clean application to conveniently download photo galleries using modern web
technologies, Java + Spring boot for the back end and angular + electron for the front end.
This is my spin for a gallery ripper app for [vipergirls](https://vipergirls.to) website.
The purpose of this project is to build a robust and clean application to conveniently download photo galleries using modern web technologies, Java + Spring boot for the back end and angular + electron for the front end.
## How to build
You need a recent version of maven 3.6.1+.
The application support 2 build modes:
- Desktop app (*Partially cross platform): Self contained app, built with electron, depends on Java runtimes.
- Server app (Fully cross platform): Depends on Java runtimes, you need a web browser to access the app UI.
- Desktop app: Self contained app, built with electron includes Java runtimes.
- Server app: Depends on Java runtimes, you need a web browser to access the app UI.
Most people will be interested only on the desktop app, however if you have a nas or a server, the server app is there
for you.
Most people will be interested only on the desktop app, however if you have a nas or a server, the server app is there for you.
To build the server app:
mvn clean install -DskipTests
mvn clean install
To build the desktop app:
mvn clean install -Pelectron -DskipTests
mvn clean install -Pelectron
Maven will automatically handle front end compilation. However, for development, you will need to install a recent
version of nodejs on your system.
Maven will automatically handle front end compilation. However, for development, you will need to install a recent version of nodejs on your system.
If you feel generous and want to support me, you can donate some coins, and I would be very grateful 🙏
I accept [PayPal](https://www.paypal.com/myaccount/transfer/homepage/buy/preview) donations, following is my email
address
dev.vripper@gmail.com
*Partially cross platform:
Technically cross platform, but the Desktop app relies on electron builder to make appropriate binaries for each platform. mac binaries can only be generated on a mac, and i don't own one. So i can only provide binaries for windows and linux.
The Server app however depends only on Java, so it will work just fine on a mac, you only need a browser to access it.
+4 -4
View File
@@ -4,17 +4,17 @@
<modelVersion>4.0.0</modelVersion>
<groupId>tn.mnlr</groupId>
<artifactId>vripper</artifactId>
<version>3.5.4</version>
<version>1.4.6-SNAPSHOT</version>
<packaging>pom</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.5</version>
<relativePath/>
<version>2.1.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<modules>
<module>vripper-server</module>
<module>vripper-ui</module>
<module>vripper-electron</module>
</modules>
</project>
</project>
Binary file not shown.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 303 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 592 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 KiB

+108 -152
View File
@@ -1,17 +1,13 @@
require('v8-compile-cache');
const {app, BrowserWindow} = require("electron");
const { app, BrowserWindow } = require("electron");
const path = require("path");
const url = require("url");
const getPort = require("get-port");
const {spawn} = require("child_process");
const {ipcMain} = require("electron");
const {dialog} = require("electron");
const { spawn } = require("child_process");
const { ipcMain } = require("electron");
const commandExists = require("command-exists").sync;
const { dialog } = require("electron");
const axios = require('axios');
const windowStateKeeper = require('electron-window-state');
// non null value when it is an AppImage
const appImageDir = process.env.APPDIR;
const appImagePath = process.env.APPIMAGE;
const appDir = process.env.APPDIR;
let win;
let vripperServer;
@@ -23,160 +19,120 @@ let terminationInteval;
let terminated = false;
process.on("uncaughtException", err => {
dialog.showErrorBox(err.message, err.stack);
process.exit(1);
dialog.showErrorBox(err.message, err.stack);
process.exit(1);
});
createWindow = () => {
if (process.platform === 'win32') {
app.setAppUserModelId('tn.mnlr.vripper');
}
let icon;
if (process.platform === 'win32') {
icon = __dirname + '/icon.ico';
} else if (process.platform === 'linux') {
icon = __dirname + '/icon.png';
}
const mainWindowState = windowStateKeeper({
defaultWidth: 1024,
defaultHeight: 800
});
win = new BrowserWindow({
x: mainWindowState.x,
y: mainWindowState.y,
width: mainWindowState.width,
height: mainWindowState.height,
minWidth: 800,
minHeight: 600,
webPreferences: {
nodeIntegration: true,
enableRemoteModule: true
},
icon: icon
});
mainWindowState.manage(win);
win.removeMenu();
win.setMenu(null);
win.setMenuBarVisibility(false);
win.loadURL(
url.format({
pathname: path.join(__dirname, `vripper-ui/index.html`),
protocol: "file:",
slashes: true
})
);
// win.webContents.openDevTools();
win.on("closed", () => {
win = null;
});
if (!commandExists("java")) {
dialog.showErrorBox(
"Java command is missing",
"Java cannot be found on your PATH, make sure to install java before running Viper Ripper"
);
process.exit(1);
}
shutdownServer = () => {
axios.post('http://localhost:' + serverPort + '/actuator/shutdown', {}, {
headers: {'content-type': 'application/json'},
}).then((response) => {
terminationInteval = setInterval(() => {
terminationAttemps++;
if (terminated) {
console.log('viper server terminated');
clearInterval(terminationInteval);
app.quit();
} else if (terminationAttemps > maxTerminationAttemps) {
console.log('viper server is not terminated');
console.log('Proceed to kill');
vripperServer.kill('SIGKILL');
clearInterval(terminationInteval);
app.quit();
}
}, 1000);
}).catch((error) => {
// Terminate immediately
console.log(error);
vripperServer.kill('SIGKILL');
terminated = true;
app.quit();
});
function createWindow() {
let icon;
if(process.platform === "win32") {
icon = path.join(__dirname, `icon.ico`);
} else {
icon = path.join(__dirname, `icon.icns`);
}
win = new BrowserWindow({
width: 1024,
height: 768,
minWidth: 640,
minHeight: 480,
frame: false,
webPreferences: {
nodeIntegration: true
},
icon: icon
});
win.removeMenu();
win.loadURL(
url.format({
pathname: path.join(__dirname, `vripper-ui/index.html`),
protocol: "file:",
slashes: true
})
);
// win.webContents.openDevTools();
win.on("closed", () => {
win = null;
});
}
getPort().then(port => {
serverPort = port;
ipcMain.on("get-port", event => {
event.reply("port", port);
});
vripperServer = spawn("java", [
"-Dvripper.server.port=" + port,
"-jar",
appDir !== undefined ? path.join(appDir, "bin/vripper-server.jar") :
path.join(app.getPath('exe'), "../bin/vripper-server.jar")
], {
stdio: 'ignore'
});
vripperServer.on('exit', (code, signal) => {
console.log(`vripper server terminated, code = ${code}, signal = ${signal}`);
terminated = true;
app.quit();
});
});
const gotTheLock = app.requestSingleInstanceLock();
if (!gotTheLock) {
app.quit();
app.quit();
} else {
getPort().then(port => {
serverPort = port;
ipcMain.on("get-port", event => {
event.reply("port", port);
});
app.on("second-instance", (event, commandLine, workingDirectory) => {
if (win) {
if (win.isMinimized()) win.restore();
win.focus();
}
});
const appPath = path.join(app.getAppPath(), '../../');
let javaBinPath, jarPath, baseDir;
if (appImageDir !== undefined && process.platform === 'linux') {
javaBinPath = path.join(appImageDir, "java-runtime/bin/java");
jarPath = path.join(appImageDir, "bin/vripper-server.jar");
baseDir = path.join(appImagePath, '..');
} else if (process.platform === 'darwin') {
javaBinPath = path.join(appPath, "java-runtime/bin/java");
jarPath = path.join(appPath, "bin/vripper-server.jar");
baseDir = app.getPath('appData');
} else if (process.platform === 'win32') {
javaBinPath = path.join(appPath, "java-runtime/bin/java");
jarPath = path.join(appPath, "bin/vripper-server.jar");
baseDir = appPath;
} else {
console.error(`Unknown platform ${process.platform}`);
app.on("ready", createWindow);
app.on("window-all-closed", () => {
if (process.platform !== "darwin") {
axios.post('http://localhost:' + serverPort + '/actuator/shutdown', {}, {
headers: { 'content-type': 'application/json' },
}).then((response) => {
terminationInteval= setInterval(() => {
terminationAttemps++;
if(terminated) {
console.log('viper server terminated');
clearInterval(terminationInteval);
app.quit();
}
vripperServer = spawn(javaBinPath, [
"-Xms256m",
"-Dvripper.server.port=" + port,
"-Dbase.dir=" + baseDir,
"-Djava.net.preferIPv4Stack=true",
"-jar",
jarPath
], {
stdio: 'ignore'
});
vripperServer.on('exit', (code, signal) => {
console.log(`vripper server terminated, code = ${code}, signal = ${signal}`);
terminated = true;
} else if(terminationAttemps > maxTerminationAttemps) {
console.log('viper server is not terminated');
console.log('Proceed to kill');
vripperServer.kill('SIGKILL');
clearInterval(terminationInteval);
app.quit();
});
});
}
}, 1000);
})
.catch((error) => {
// Terminate immediately
vripperServer.kill('SIGKILL');
terminated = true;
app.quit();
});
}
});
app.on("second-instance", (event, commandLine, workingDirectory) => {
if (win) {
if (win.isMinimized()) win.restore();
win.focus();
}
});
app.on("ready", createWindow);
app.on("window-all-closed", () => {
if (process.platform !== "darwin") {
shutdownServer();
}
});
app.on("will-quit", () => {
if (process.platform === "darwin") {
shutdownServer();
}
});
app.on("activate", () => {
if (win === null) {
createWindow();
}
});
app.on("activate", () => {
if (win === null) {
createWindow();
}
});
}
+3269
View File
File diff suppressed because it is too large Load Diff
+16 -39
View File
@@ -1,14 +1,12 @@
{
"name": "vripper-electron",
"version": "3.5.4",
"description": "A ripper for vipergirls.to built using web technolgies",
"version": "1.4.6-SNAPSHOT",
"description": "",
"main": "main.js",
"author": "death-claw <53543762+death-claw@users.noreply.github.com>",
"homepage": "https://github.com/death-claw/vripper-project",
"author": "",
"license": "ISC",
"build": {
"appId": "tn.mnlr.vripper",
"productName": "VRipper",
"files": [
"**/*",
"!**/node_modules/*/{CHANGELOG.md,README.md,README,readme.md,readme}",
@@ -22,10 +20,6 @@
"!**/{__pycache__,thumbs.db,.flowconfig,.idea,.vs,.nyc_output}",
"!**/{appveyor.yml,.travis.yml,circle.yml}",
"!**/{npm-debug.log,yarn.lock,.yarn-integrity,.yarn-metadata.json}",
"!**/build-dir",
"!**/java-runtime",
"!**/pre-build.js",
"!**/pom.xml",
{
"from": "./build/",
"to": "."
@@ -35,50 +29,33 @@
{
"from": "../vripper-server/target/vripper-server-${version}-electron.jar",
"to": "bin/vripper-server.jar"
},
{
"from": "java-runtime",
"to": "java-runtime"
}
],
"win": {
"icon": "icon.ico",
"target": [
"dir"
]
"target": "nsis"
},
"linux": {
"synopsis": "vipergirls.to ripper",
"category": "Utility",
"packageCategory": "Utility",
"target": [
"AppImage"
"zip"
]
},
"mac": {
"category": "public.app-category.utilities",
"target": [
"dmg"
],
"icon": "icon.icns"
}
},
"scripts": {
"start": "electron .",
"dist": "node pre-build.js && electron-builder"
"dist": "node pre-build.js && build"
},
"devDependencies": {
"electron": "11.1.1",
"electron-builder": "22.9.1"
"electron": "^5.0.5",
"electron-builder": "^20.44.4"
},
"dependencies": {
"axios": "0.21.1",
"cheerio": "1.0.0-rc.3",
"copy-dir": "1.3.0",
"electron-context-menu": "2.1.0",
"electron-window-state": "^5.0.3",
"get-port": "5.1.1",
"rimraf": "3.0.2",
"v8-compile-cache": "2.1.1"
"axios": "^0.19.0",
"cheerio": "^1.0.0-rc.3",
"command-exists": "^1.2.8",
"copy-dir": "^1.1.0",
"custom-electron-titlebar": "^3.0.10",
"electron-context-menu": "^0.13.0",
"get-port": "^5.0.0",
"rimraf": "^2.6.3"
}
}
+71 -71
View File
@@ -1,77 +1,77 @@
<?xml version="1.0"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>tn.mnlr</groupId>
<artifactId>vripper</artifactId>
<version>3.5.4</version>
</parent>
<artifactId>vripper-electron</artifactId>
<name>vripper-electron</name>
<build>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>tn.mnlr</groupId>
<artifactId>vripper</artifactId>
<version>1.4.6-SNAPSHOT</version>
</parent>
<artifactId>vripper-electron</artifactId>
<name>vripper-electron</name>
<build>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<classifier>${buildClassifier}</classifier>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>electron</id>
<properties>
<buildClassifier>electron</buildClassifier>
</properties>
<dependencies>
<dependency>
<groupId>tn.mnlr</groupId>
<artifactId>vripper-server</artifactId>
<version>${project.version}</version>
<scope>runtime</scope>
<classifier>${buildClassifier}</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.3</version>
<configuration>
<nodeVersion>v12.2.0</nodeVersion>
<npmVersion>6.9.0</npmVersion>
<workingDirectory>build-dir</workingDirectory>
</configuration>
<executions>
<execution>
<id>install node and npm</id>
<goals>
<goal>install-node-and-npm</goal>
</goals>
</execution>
<execution>
<id>npm install</id>
<goals>
<goal>npm</goal>
</goals>
</execution>
<execution>
<id>npm run build</id>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<classifier>${buildClassifier}</classifier>
<arguments>run dist</arguments>
</configuration>
</plugin>
<phase>generate-resources</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>electron</id>
<properties>
<buildClassifier>electron</buildClassifier>
</properties>
<dependencies>
<dependency>
<groupId>tn.mnlr</groupId>
<artifactId>vripper-server</artifactId>
<version>${project.version}</version>
<scope>runtime</scope>
<classifier>${buildClassifier}</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.11.2</version>
<configuration>
<nodeVersion>v14.16.0</nodeVersion>
<yarnVersion>v1.22.10</yarnVersion>
<workingDirectory>build-dir</workingDirectory>
</configuration>
<executions>
<execution>
<id>install node and yarn</id>
<goals>
<goal>install-node-and-yarn</goal>
</goals>
</execution>
<execution>
<id>yarn install</id>
<goals>
<goal>yarn</goal>
</goals>
</execution>
<execution>
<id>yarn run build</id>
<goals>
<goal>yarn</goal>
</goals>
<configuration>
<arguments>run dist</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</build>
</profile>
</profiles>
</project>
+1 -8
View File
@@ -2,7 +2,7 @@ const cheerio = require('cheerio');
const fs = require('fs');
const rimraf = require("rimraf");
const copydir = require("copy-dir");
const {execSync} = require("child_process");
rimraf.sync('./build/vripper-ui');
copydir.sync('../vripper-ui/dist/vripper-ui', './build/vripper-ui', {});
@@ -16,10 +16,3 @@ $('body').prepend(`<script>require('../renderer.js')</script>`);
fs.writeFileSync('./build/vripper-ui/index.html', $.html());
console.log('Building runtime environment');
rimraf.sync('java-runtime');
execSync('jlink --no-header-files --no-man-pages --compress=2 --strip-debug --add-modules java.base,java.desktop,java.instrument,java.management,java.security.jgss,java.sql,jdk.unsupported,jdk.crypto.ec --output java-runtime');
if (process.platform === 'linux') {
console.log('Stripping libjvm.so');
execSync('strip -p --strip-unneeded java-runtime/lib/server/libjvm.so');
}
+10 -1
View File
@@ -1,3 +1,12 @@
console.log("renderer intialized");
const customTitlebar = require("custom-electron-titlebar");
const electron = require("electron").remote;
const contextMenu = require("electron-context-menu");
contextMenu({});
contextMenu({});
const menu = new electron.Menu();
const titlebar = new customTitlebar.Titlebar({
backgroundColor: customTitlebar.Color.fromHex("#3f51b5"),
menu: menu
});
titlebar.updateTitle("Viper Ripper");
-12
View File
@@ -1,12 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
<output url="file://$MODULE_DIR$/target/classes" />
<output-test url="file://$MODULE_DIR$/target/test-classes" />
<content url="file://$MODULE_DIR$">
<excludeFolder url="file://$MODULE_DIR$/target" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>
File diff suppressed because it is too large Load Diff
+63 -29
View File
@@ -1,21 +1,29 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>tn.mnlr</groupId>
<artifactId>vripper</artifactId>
<version>3.5.4</version>
<version>1.4.6-SNAPSHOT</version>
</parent>
<artifactId>vripper-server</artifactId>
<name>vripper-server</name>
<description>vripper-server</description>
<properties>
<java.version>11</java.version>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jersey</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
@@ -24,19 +32,6 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
</dependency>
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
@@ -55,25 +50,23 @@
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.6</version>
</dependency>
<dependency>
<groupId>net.sourceforge.htmlcleaner</groupId>
<artifactId>htmlcleaner</artifactId>
<version>2.24</version>
<version>2.22</version>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<groupId>io.reactivex.rxjava2</groupId>
<artifactId>rxjava</artifactId>
<version>2.2.6</version>
</dependency>
<dependency>
<groupId>net.jodah</groupId>
<artifactId>failsafe</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
</dependency>
<dependency>
<groupId>tn.mnlr</groupId>
<artifactId>vripper-ui</artifactId>
@@ -89,12 +82,6 @@
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<classifier>${buildClassifier}</classifier>
<archive>
<manifestEntries>
<Build-Time>${maven.build.timestamp}</Build-Time>
<Built-By>${user.name}</Built-By>
</manifestEntries>
</archive>
</configuration>
</plugin>
<plugin>
@@ -112,6 +99,53 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>get-the-git-infos</id>
<goals>
<goal>revision</goal>
</goals>
</execution>
<execution>
<id>validate-the-git-infos</id>
<goals>
<goal>validateRevision</goal>
</goals>
</execution>
</executions>
<configuration>
<dotGitDirectory>${project.parent.basedir}/.git</dotGitDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<archive>
<manifestEntries>
<Build-Time>${maven.build.timestamp}</Build-Time>
</manifestEntries>
<manifestSections>
<manifestSection>
<name>Versions</name>
<manifestEntries>
<Implementation-Version>${git.build.version}</Implementation-Version>
<Implementation-SCM-Revision>${git.commit.id}</Implementation-SCM-Revision>
<Implementation-SCM-Branch>${git.branch}</Implementation-SCM-Branch>
<Implementation-SCM-Time>${git.build.time}</Implementation-SCM-Time>
<Implementation-SCM-Dirty>${git.dirty}</Implementation-SCM-Dirty>
<Implementation-SCM-Tags>${git.tags}</Implementation-SCM-Tags>
</manifestEntries>
</manifestSection>
</manifestSections>
</archive>
</configuration>
</plugin>
</plugins>
</build>
@@ -1,31 +0,0 @@
package tn.mnlr.vripper;
import lombok.Getter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
import tn.mnlr.vripper.jpa.repositories.IPostRepository;
import tn.mnlr.vripper.services.DataService;
@Component
public class EventListenerBean {
@Getter private static boolean init = false;
private final IPostRepository postRepository;
private final DataService dataService;
@Autowired
public EventListenerBean(IPostRepository postRepository, DataService dataService) {
this.postRepository = postRepository;
this.dataService = dataService;
}
@EventListener
public void onApplicationEvent(ContextRefreshedEvent event) {
postRepository.setDownloadingToStopped();
dataService.sortPostsByRank();
init = true;
}
}
@@ -1,6 +1,7 @@
package tn.mnlr.vripper;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
@@ -10,30 +11,37 @@ import org.springframework.stereotype.Component;
import java.util.Map;
@Component
@Slf4j
public class SpringContext implements ApplicationContextAware {
private static ConfigurableApplicationContext context;
private static final Logger logger = LoggerFactory.getLogger(SpringContext.class);
public static <T> T getBean(Class<T> beanClass) {
return context.getBean(beanClass);
}
private static ConfigurableApplicationContext context;
public static <T> Map<String, T> getBeansOfType(Class<T> beanClass) {
return context.getBeansOfType(beanClass);
}
public static void close() {
log.info("Application terminating...");
if (context != null) {
context.close();
/**
* Returns the Spring managed bean instance of the given class type (if it exists).
* Returns null otherwise.
*
* @param beanClass
* @return
*/
public static <T extends Object> T getBean(Class<T> beanClass) {
return context.getBean(beanClass);
}
}
@Override
public void setApplicationContext(ApplicationContext context) throws BeansException {
public static <T extends Object> Map<String, T> getBeansOfType(Class<T> beanClass) {
return context.getBeansOfType(beanClass);
}
// store ApplicationContext reference to access required beans later on
SpringContext.context = ((ConfigurableApplicationContext) context);
}
}
public static void close() {
logger.info("Application terminating...");
context.close();
}
@Override
public void setApplicationContext(ApplicationContext context) throws BeansException {
// store ApplicationContext reference to access required beans later on
SpringContext.context = ((ConfigurableApplicationContext) context);
}
}
@@ -1,24 +0,0 @@
package tn.mnlr.vripper;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
@Slf4j
public class Utils {
public static String throwableToString(Throwable th) {
try (StringWriter stringWriter = new StringWriter();
PrintWriter printWriter = new PrintWriter(stringWriter)) {
th.printStackTrace(printWriter);
printWriter.flush();
stringWriter.flush();
return stringWriter.toString();
} catch (IOException e) {
log.error("Failed to convert throwable to string", e);
}
return "Cannot display error, please check the log file for details";
}
}
@@ -1,19 +1,65 @@
package tn.mnlr.vripper;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Component;
import tn.mnlr.vripper.exception.VripperException;
import tn.mnlr.vripper.services.AppSettingsService;
import tn.mnlr.vripper.services.PersistenceService;
import tn.mnlr.vripper.services.VipergirlsAuthService;
import java.io.File;
@SpringBootApplication
@Slf4j
public class VripperApplication {
public static void main(String[] args) {
try {
Runtime.getRuntime().addShutdownHook(new Thread(SpringContext::close));
SpringApplication.run(VripperApplication.class, args);
} catch (Exception e) {
log.error("Failed to run the application", e);
private static final Logger logger = LoggerFactory.getLogger(VripperApplication.class);
public static final String dataPath = System.getProperty("vripper.datapath", ".") + File.separator + "data.json";
public static void main(String[] args) {
try {
SpringApplication.run(VripperApplication.class, args);
} catch (Exception e) {
logger.error("Failed to run the application", e);
}
}
@Component
public class AppCommandRunner implements CommandLineRunner {
@Autowired
private VipergirlsAuthService authService;
@Autowired
private PersistenceService persistenceService;
@Autowired
private AppSettingsService appSettingsService;
@Override
public void run(String... args) {
persistenceService.restore();
appSettingsService.restore();
registerShutdownHook();
try {
authService.authenticate();
} catch (VripperException e) {
logger.error("Cannot authenticate user with ViperGirls", e);
}
}
private void registerShutdownHook() {
Runtime.getRuntime().addShutdownHook(new Thread(SpringContext::close));
}
}
}
}
@@ -1,295 +0,0 @@
package tn.mnlr.vripper.download;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import net.jodah.failsafe.function.CheckedRunnable;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.AbstractExecutionAwareRequest;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.util.EntityUtils;
import tn.mnlr.vripper.SpringContext;
import tn.mnlr.vripper.exception.DownloadException;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.jpa.domain.Image;
import tn.mnlr.vripper.jpa.domain.Post;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.services.*;
import tn.mnlr.vripper.services.domain.Settings;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.util.*;
@Slf4j
public class DownloadJob implements CheckedRunnable {
private static final Object LOCK = new Object();
private static final int READ_BUFFER_SIZE = 8192;
private final DataService dataService;
private final PathService pathService;
private final ConnectionService cm;
private final VGAuthService authService;
private final DownloadSpeedService downloadSpeedService;
private final Settings settings;
private final HttpClientContext context;
@Getter private final Image image;
@Getter private final Post post;
private volatile boolean stopped = false;
@Getter private volatile boolean finished = false;
DownloadJob(Post post, Image image, Settings settings) {
this.image = image;
this.post = post;
this.settings = settings;
dataService = SpringContext.getBean(DataService.class);
pathService = SpringContext.getBean(PathService.class);
cm = SpringContext.getBean(ConnectionService.class);
authService = SpringContext.getBean(VGAuthService.class);
downloadSpeedService = SpringContext.getBean(DownloadSpeedService.class);
context = HttpClientContext.create();
context.setCookieStore(new BasicCookieStore());
context.setAttribute(
ContextAttributes.OPEN_CONNECTION.toString(),
Collections.synchronizedList(new ArrayList<AbstractExecutionAwareRequest>()));
}
public void download(final Post post, final Image image) throws DownloadException {
try {
image.setStatus(Status.DOWNLOADING);
image.setCurrent(0);
dataService.updateImageStatus(image.getStatus(), image.getId());
dataService.updateImageCurrent(image.getCurrent(), image.getId());
synchronized (LOCK) {
if (!post.getStatus().equals(Status.DOWNLOADING)
&& !post.getStatus().equals(Status.PARTIAL)) {
post.setStatus(Status.DOWNLOADING);
dataService.updatePostStatus(post.getStatus(), post.getId());
}
// The post may be updated and the download directory might be set by another thread
Post updatedPost = dataService.findById(post.getId()).orElseThrow();
if (updatedPost.getDownloadDirectory() == null) {
pathService.createDefaultPostFolder(updatedPost, settings);
}
if (settings.getLeaveThanksOnStart() != null && settings.getLeaveThanksOnStart()) {
authService.leaveThanks(updatedPost);
}
}
if (stopped) {
return;
}
/*
* HOST SPECIFIC
*/
log.debug(
String.format(
"Getting image url and name from %s using %s", image.getUrl(), image.getHost()));
HostService.NameUrl nameAndUrl = image.getHost().getNameAndUrl(image.getUrl(), context);
log.debug(String.format("Resolved name for %s: %s", image.getUrl(), nameAndUrl.getName()));
log.debug(
String.format("Resolved image url for %s: %s", image.getUrl(), nameAndUrl.getUrl()));
/*
* END HOST SPECIFIC
*/
if (stopped) {
return;
}
String formatImageFileName = pathService.formatImageFileName(nameAndUrl.getName());
log.debug(
String.format(
"Sanitizing image name from %s to %s", nameAndUrl.getName(), formatImageFileName));
nameAndUrl = new HostService.NameUrl(formatImageFileName, nameAndUrl.getUrl());
HttpClient client = cm.getClient().build();
log.debug(String.format("Downloading %s", nameAndUrl.getUrl()));
HttpGet httpGet = cm.buildHttpGet(nameAndUrl.getUrl(), context);
httpGet.addHeader("Referer", image.getUrl());
try (CloseableHttpResponse response =
(CloseableHttpResponse) client.execute(httpGet, context)) {
if (response.getStatusLine().getStatusCode() / 100 != 2) {
EntityUtils.consumeQuietly(response.getEntity());
throw new DownloadException(
String.format("Server returned code %d", response.getStatusLine().getStatusCode()));
}
if (stopped) {
return;
}
File outputFile = Files.createTempFile("vripper", "tmp").toFile();
try (InputStream downloadStream = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream(outputFile)) {
if (stopped) {
return;
}
image.setTotal(response.getEntity().getContentLength());
dataService.updateImageTotal(image.getTotal(), image.getId());
log.debug(String.format("%s length is %d", nameAndUrl.getUrl(), image.getTotal()));
log.debug(String.format("Starting data transfer for %s", nameAndUrl.getUrl()));
byte[] buffer = new byte[READ_BUFFER_SIZE];
int read;
while ((read = downloadStream.read(buffer, 0, READ_BUFFER_SIZE)) != -1 && !stopped) {
fos.write(buffer, 0, read);
image.increase(read);
downloadSpeedService.increase(read);
dataService.updateImageCurrent(image.getCurrent(), image.getId());
}
fos.flush();
EntityUtils.consumeQuietly(response.getEntity());
if (stopped) {
return;
}
}
checkImageTypeAndRename(
dataService.findById(post.getId()).orElseThrow(),
outputFile,
nameAndUrl.getName(),
image.getIndex());
}
} catch (Exception e) {
if (stopped) {
return;
}
throw new DownloadException(e);
} finally {
if (image.getCurrent() == image.getTotal() && image.getTotal() > 0) {
image.setStatus(Status.COMPLETE);
} else if (stopped) {
image.setStatus(Status.STOPPED);
} else {
image.setStatus(Status.ERROR);
}
dataService.updateImageStatus(image.getStatus(), image.getId());
finished = true;
}
}
private void checkImageTypeAndRename(Post post, File outputFile, String imageName, int index)
throws HostException {
try (ImageInputStream iis = ImageIO.createImageInputStream(outputFile)) {
Iterator<ImageReader> it = ImageIO.getImageReaders(iis);
if (!it.hasNext()) {
throw new HostException("Image file is not recognized!");
}
ImageReader reader = it.next();
if (reader.getFormatName().equalsIgnoreCase("JPEG")) {
String imageNameLC = imageName.toLowerCase();
if (!imageNameLC.endsWith("_jpg") && !imageNameLC.endsWith("_jpeg")) {
imageName += ".jpg";
} else {
String toReplace = null;
if (imageNameLC.endsWith("_jpg")) {
toReplace = "_jpg";
} else if (imageNameLC.endsWith("_jpeg")) {
toReplace = "_jpeg";
}
if (toReplace != null) {
imageName = imageName.substring(0, imageName.length() - toReplace.length()) + ".jpg";
}
}
} else if (reader.getFormatName().equalsIgnoreCase("PNG")) {
String imageNameLC = imageName.toLowerCase();
if (!imageNameLC.endsWith("_png")) {
imageName += ".png";
} else {
String toReplace = null;
if (imageNameLC.endsWith("_png")) {
toReplace = "_png";
}
if (toReplace != null) {
imageName = imageName.substring(0, imageName.length() - toReplace.length()) + ".png";
}
}
}
} catch (Exception e) {
throw new HostException("Failed to guess image format", e);
}
try {
pathService.getDirectoryAccess().lock();
File downloadDestinationFolder = pathService.calcDownloadDirectory(post, settings);
File outImage =
new File(
downloadDestinationFolder,
(settings.getForceOrder() ? String.format("%03d_", index) : "") + imageName);
Files.copy(outputFile.toPath(), outImage.toPath(), StandardCopyOption.REPLACE_EXISTING);
} catch (Exception e) {
throw new HostException("Failed to rename the image", e);
} finally {
pathService.getDirectoryAccess().unlock();
try {
Files.delete(outputFile.toPath());
} catch (IOException e) {
log.warn(String.format("Failed to delete temporary file %s", outputFile.getAbsolutePath()));
}
}
}
@Override
public void run() throws Exception {
if (stopped) {
finished = true;
return;
}
log.debug(String.format("Starting downloading %s", image.getUrl()));
download(post, image);
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
DownloadJob that = (DownloadJob) o;
return Objects.equals(image, that.image) && Objects.equals(post, that.post);
}
@Override
public int hashCode() {
return Objects.hash(image, post);
}
public void stop() {
this.stopped = true;
List<AbstractExecutionAwareRequest> requests =
(List<AbstractExecutionAwareRequest>)
this.context.getAttribute(ContextAttributes.OPEN_CONNECTION.toString());
if (requests != null) {
for (AbstractExecutionAwareRequest request : requests) {
request.abort();
}
}
}
public enum ContextAttributes {
OPEN_CONNECTION("OPEN_CONNECTION");
private final String value;
ContextAttributes(String value) {
this.value = value;
}
@Override
public String toString() {
return value;
}
}
}
@@ -1,70 +0,0 @@
package tn.mnlr.vripper.download;
import lombok.extern.slf4j.Slf4j;
import net.jodah.failsafe.Failsafe;
import tn.mnlr.vripper.SpringContext;
import tn.mnlr.vripper.Utils;
import tn.mnlr.vripper.jpa.domain.LogEvent;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.jpa.repositories.ILogEventRepository;
import tn.mnlr.vripper.services.ConnectionService;
import tn.mnlr.vripper.services.DataService;
import java.time.LocalDateTime;
@Slf4j
public class DownloadJobWrapper implements Runnable {
private final DownloadService downloadService;
private final DataService dataService;
private final ConnectionService connectionService;
private final ILogEventRepository eventRepository;
private final DownloadJob downloadJob;
public DownloadJobWrapper(final DownloadJob downloadJob) {
downloadService = SpringContext.getBean(DownloadService.class);
dataService = SpringContext.getBean(DataService.class);
connectionService = SpringContext.getBean(ConnectionService.class);
eventRepository = SpringContext.getBean(ILogEventRepository.class);
this.downloadJob = downloadJob;
}
@Override
public void run() {
Failsafe.with(connectionService.getRetryPolicy())
.onFailure(
e -> {
try {
LogEvent logEvent =
new LogEvent(
LogEvent.Type.DOWNLOAD,
LogEvent.Status.ERROR,
LocalDateTime.now(),
String.format(
"Failed to download %s\n %s",
downloadJob.getImage().getUrl(),
Utils.throwableToString(e.getFailure())));
eventRepository.save(logEvent);
} catch (Exception exp) {
log.error("Failed to save event", exp);
}
log.error(
String.format(
"Failed to download %s after %d tries",
downloadJob.getImage().getUrl(), e.getAttemptCount()),
e.getFailure());
downloadJob.getImage().setStatus(Status.ERROR);
dataService.updateImageStatus(
downloadJob.getImage().getStatus(), downloadJob.getImage().getId());
})
.onComplete(
e -> {
dataService.afterJobFinish(downloadJob.getImage(), downloadJob.getPost());
downloadService.afterJobFinish(downloadJob);
log.debug(String.format("Finished downloading %s", downloadJob.getImage().getUrl()));
})
.run(downloadJob);
}
}
@@ -1,277 +0,0 @@
package tn.mnlr.vripper.download;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.jpa.domain.Image;
import tn.mnlr.vripper.jpa.domain.Post;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.services.DataService;
import tn.mnlr.vripper.services.MetadataService;
import tn.mnlr.vripper.services.SettingsService;
import tn.mnlr.vripper.services.domain.Settings;
import javax.annotation.PostConstruct;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
@Service
@Slf4j
public class DownloadService {
private final int MAX_POOL_SIZE = 12;
private final ConcurrentHashMap<Host, AtomicInteger> threadCount = new ConcurrentHashMap<>();
private final ExecutorService executor = Executors.newFixedThreadPool(MAX_POOL_SIZE);
private final List<DownloadJob> running = new ArrayList<>();
private final List<DownloadJob> pending = new ArrayList<>();
private final SettingsService settingsService;
private final DataService dataService;
private final MetadataService metadataService;
private final List<Host> hosts;
private Thread pollThread;
@Autowired
public DownloadService(
SettingsService settingsService,
DataService dataService,
MetadataService metadataService,
List<Host> hosts) {
this.settingsService = settingsService;
this.dataService = dataService;
this.metadataService = metadataService;
this.hosts = hosts;
}
@PostConstruct
private void init() {
pollThread = new Thread(this::start, "Polling thread");
pollThread.start();
}
public void destroy() throws Exception {
log.info("Shutting down ExecutionService");
pollThread.interrupt();
executor.shutdown();
dataService
.findAllPosts()
.forEach(
p -> {
log.debug(String.format("Stopping download jobs for %s", p));
this.stopRunning(p.getPostId());
});
executor.awaitTermination(5, TimeUnit.SECONDS);
}
private synchronized void stopRunning(@NonNull String postId) {
List<DownloadJob> stopping = new ArrayList<>();
Iterator<DownloadJob> iterator = running.iterator();
while (iterator.hasNext()) {
DownloadJob downloadJob = iterator.next();
if (postId.equals(downloadJob.getPost().getPostId())) {
downloadJob.stop();
iterator.remove();
stopping.add(downloadJob);
}
}
while (!stopping.isEmpty()) {
stopping.removeIf(DownloadJob::isFinished);
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
public void stopAll(List<String> postIds) {
stop(
Objects.requireNonNullElseGet(
postIds,
() ->
dataService.findAllPosts().stream()
.map(Post::getPostId)
.collect(Collectors.toList())));
}
public void restartAll(List<String> posIds) {
restart(
Objects.requireNonNullElseGet(
posIds,
() ->
dataService.findAllPosts().stream()
.map(Post::getPostId)
.collect(Collectors.toList())));
}
private void restart(@NonNull List<String> postIds) {
Map<Post, Collection<Image>> data = new HashMap<>();
for (String postId : postIds) {
if (isPending(postId)) {
log.warn(
String.format("Cannot restart, jobs are currently running for post id %s", postIds));
continue;
}
List<Image> images = dataService.findByPostIdAndIsNotCompleted(postId);
if (images.isEmpty()) {
continue;
}
Post post = dataService.findPostByPostId(postId).orElseThrow();
log.debug(String.format("Restarting %d jobs for post id %s", images.size(), postIds));
data.put(post, images);
}
enqueue(data);
}
private synchronized boolean isPending(String postId) {
return pending.stream().anyMatch(p -> p.getPost().getPostId().equals(postId));
}
private synchronized boolean isRunning(String postId) {
return running.stream().anyMatch(p -> p.getPost().getPostId().equals(postId));
}
private synchronized void stop(List<String> postIds) {
for (String postId : postIds) {
final Post post = dataService.findPostByPostId(postId).orElseThrow();
if (post == null) {
continue;
}
pending.removeIf(p -> p.getPost().equals(post));
stopRunning(postId);
dataService.stopImagesByPostIdAndIsNotCompleted(postId);
dataService.finishPost(post);
}
metadataService.stopFetchingMetadata(postIds);
}
private boolean canRun(Host host) {
boolean canRun;
int totalRunning = threadCount.values().stream().mapToInt(AtomicInteger::get).sum();
canRun =
threadCount.get(host).get() < settingsService.getSettings().getMaxThreads()
&& (settingsService.getSettings().getMaxTotalThreads() == 0
? totalRunning < MAX_POOL_SIZE
: totalRunning < settingsService.getSettings().getMaxTotalThreads());
if (canRun) {
threadCount.get(host).incrementAndGet();
return true;
}
return false;
}
private Map<Host, Integer> candidateCount() {
HashMap<Host, Integer> map = new HashMap<>();
hosts.forEach(
h -> {
AtomicInteger count = threadCount.get(h);
if (count == null) {
count = new AtomicInteger(0);
threadCount.put(h, count);
}
map.put(h, settingsService.getSettings().getMaxThreads() - count.get());
});
return map;
}
private List<DownloadJob> getCandidates(Map<Host, Integer> candidateCount, int max) {
int i = 0;
List<DownloadJob> candidates = new ArrayList<>();
for (DownloadJob downloadJob : pending) {
Host host = downloadJob.getImage().getHost();
Integer maxPerHost = candidateCount.get(host);
if (maxPerHost > 0 && i < max) {
candidates.add(downloadJob);
candidateCount.put(host, maxPerHost - 1);
i++;
}
if (i >= max) {
break;
}
}
return candidates;
}
public void enqueue(Map<Post, Collection<Image>> images) {
synchronized (this) {
for (Map.Entry<Post, Collection<Image>> entry : images.entrySet()) {
entry.getKey().setStatus(Status.PENDING);
dataService.updatePostStatus(entry.getKey().getStatus(), entry.getKey().getId());
for (Image image : entry.getValue()) {
log.debug(String.format("Enqueuing a job for %s", image.getUrl()));
image.init();
dataService.updateImageStatus(image.getStatus(), image.getId());
dataService.updateImageCurrent(image.getCurrent(), image.getId());
DownloadJob downloadJob =
new DownloadJob(
entry.getKey(), image, (Settings) settingsService.getSettings().clone());
pending.add(downloadJob);
}
}
pending.sort(Comparator.comparing(e -> e.getPost().getAddedOn()));
this.notify();
}
}
private void start() {
while (!Thread.interrupted()) {
try {
synchronized (this) {
List<DownloadJob> accepted = new ArrayList<>();
List<DownloadJob> candidates = getCandidates(candidateCount(), MAX_POOL_SIZE);
candidates.forEach(
c -> {
if (canRun(c.getImage().getHost())) {
accepted.add(c);
}
});
pending.removeAll(accepted);
accepted.forEach(this::push);
accepted.clear();
this.wait();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
}
private synchronized void push(DownloadJob downloadJob) {
log.debug(String.format("Scheduling a job for %s", downloadJob.getImage().getUrl()));
executor.execute(new DownloadJobWrapper(downloadJob));
running.add(downloadJob);
}
public void afterJobFinish(DownloadJob downloadJob) {
synchronized (this) {
running.remove(downloadJob);
threadCount.get(downloadJob.getImage().getHost()).decrementAndGet();
if (!isPending(downloadJob.getPost().getPostId())
&& !isRunning(downloadJob.getPost().getPostId())) {
dataService.finishPost(downloadJob.getPost());
}
this.notify();
}
}
public int pendingCount() {
return pending.size();
}
public int runningCount() {
return running.size();
}
}
@@ -0,0 +1,118 @@
package tn.mnlr.vripper.entities;
import io.reactivex.disposables.Disposable;
import io.reactivex.processors.BehaviorProcessor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.services.AppStateService;
import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
@Getter
@NoArgsConstructor
public class Image {
@Setter
private AppStateService appStateService;
private Disposable subscription;
private final String type = "img";
private String postId;
private String postName;
private Host host;
private String url;
private AtomicLong current = new AtomicLong(0);
private Status status;
private BehaviorProcessor<Image> imageStateProcessor;
@Setter
private long total = 0;
public Image(String url, String postId, String postName, Host host, AppStateService appStateService) {
this.url = url;
this.postId = postId;
this.postName = postName;
this.host = host;
this.appStateService = appStateService;
status = Status.PENDING;
imageStateProcessor = BehaviorProcessor.create();
appStateService.getCurrentImages().put(this.url, this);
appStateService.getLiveImageUpdates().onNext(this);
init();
}
public void setStatus(Status status) {
this.status = status;
update();
}
public boolean isCompleted() {
return status.equals(Status.COMPLETE);
}
public void init() {
if (imageStateProcessor != null) {
imageStateProcessor.onComplete();
}
if (subscription != null) {
subscription.dispose();
}
imageStateProcessor = BehaviorProcessor.create();
subscription = imageStateProcessor
.onBackpressureLatest()
.doOnNext(appStateService::onImageUpdate)
.subscribe();
current.set(0);
status = Status.PENDING;
imageStateProcessor.onNext(this);
}
public void setCurrent(int current) {
this.current.set(current);
update();
}
public void increase(int read) {
current.addAndGet(read);
update();
}
private void update() {
if(imageStateProcessor == null) {
return;
}
imageStateProcessor.onNext(this);
if (isCompleted()) {
imageStateProcessor.onComplete();
subscription.dispose();
imageStateProcessor = null;
}
}
public enum Status {
PENDING, DOWNLOADING, COMPLETE, ERROR, STOPPED
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Image image = (Image) o;
return url.equals(image.url);
}
@Override
public int hashCode() {
return Objects.hash(url);
}
}
@@ -0,0 +1,91 @@
package tn.mnlr.vripper.entities;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import tn.mnlr.vripper.services.AppStateService;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicInteger;
@Getter
@ToString
@NoArgsConstructor
public class Post {
@Setter
private AppStateService appStateService;
private Status status;
private final String type = "post";
private String postId;
private String title;
private String postCounter;
private String url;
private List<Image> images;
private Map<String, String> metadata;
private AtomicInteger done = new AtomicInteger(0);
private int total;
private boolean removed = false;
public Post(String title, String url, List<Image> images, Map<String, String> metadata, String postId, String postCounter, AppStateService appStateService) {
this.title = title;
this.url = url;
this.images = images;
this.metadata = metadata;
this.postId = postId;
this.postCounter = postCounter;
this.appStateService = appStateService;
total = images.size();
status = Status.PENDING;
appStateService.getCurrentPosts().put(postId, this);
appStateService.getLivePostsState().onNext(this);
}
public void setRemoved(boolean removed) {
this.removed = removed;
appStateService.getLivePostsState().onNext(this);
}
public void increase() {
done.incrementAndGet();
appStateService.getLivePostsState().onNext(this);
}
public void setStatus(Status status) {
this.status = status;
if (appStateService != null) {
appStateService.getLivePostsState().onNext(this);
}
}
public enum Status {
PENDING, DOWNLOADING, COMPLETE, ERROR, PARTIAL, STOPPED
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Post post = (Post) o;
return postId.equals(post.postId);
}
@Override
public int hashCode() {
return Objects.hash(postId);
}
}
@@ -0,0 +1,12 @@
package tn.mnlr.vripper.entities.mixin.persistance;
import com.fasterxml.jackson.databind.util.StdConverter;
import tn.mnlr.vripper.host.Host;
public class HostToString extends StdConverter<Host, String> {
@Override
public String convert(Host value) {
return value.getClass().getSimpleName();
}
}
@@ -0,0 +1,32 @@
package tn.mnlr.vripper.entities.mixin.persistance;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import io.reactivex.disposables.Disposable;
import io.reactivex.processors.BehaviorProcessor;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.services.AppStateService;
public abstract class ImagePersistanceMixin {
@JsonSerialize(converter = HostToString.class)
@JsonDeserialize(converter = StringToHost.class)
private Host host;
@JsonIgnore
private BehaviorProcessor<Image> imageStateProcessor;
@JsonIgnore
private Disposable subscription;
@JsonIgnore
private String type;
@JsonIgnore
public abstract boolean isCompleted();
@JsonIgnore
private AppStateService appStateService;
}
@@ -0,0 +1,17 @@
package tn.mnlr.vripper.entities.mixin.persistance;
import com.fasterxml.jackson.annotation.JsonIgnore;
import tn.mnlr.vripper.services.AppStateService;
public abstract class PostPersistanceMixin {
@JsonIgnore
private String type;
@JsonIgnore
private AppStateService appStateService;
@JsonIgnore
private boolean removed;
}
@@ -0,0 +1,17 @@
package tn.mnlr.vripper.entities.mixin.persistance;
import com.fasterxml.jackson.databind.util.StdConverter;
import tn.mnlr.vripper.SpringContext;
import tn.mnlr.vripper.host.Host;
import java.util.Collection;
public class StringToHost extends StdConverter<String, Host> {
private Collection<Host> hosts = SpringContext.getBeansOfType(Host.class).values();
@Override
public Host convert(String value) {
return hosts.stream().filter(e -> e.getClass().getSimpleName().equals(value)).findAny().orElse(null);
}
}
@@ -0,0 +1,23 @@
package tn.mnlr.vripper.entities.mixin.ui;
import com.fasterxml.jackson.annotation.JsonIgnore;
import io.reactivex.disposables.Disposable;
import io.reactivex.processors.BehaviorProcessor;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.services.AppStateService;
public abstract class ImageUIMixin {
@JsonIgnore
private Host host;
@JsonIgnore
private BehaviorProcessor<Image> imageStateProcessor;
@JsonIgnore
private Disposable subscription;
@JsonIgnore
private AppStateService appStateService;
}
@@ -0,0 +1,20 @@
package tn.mnlr.vripper.entities.mixin.ui;
import com.fasterxml.jackson.annotation.JsonIgnore;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.services.AppStateService;
import java.util.List;
import java.util.Map;
public abstract class PostUIMixin {
@JsonIgnore
private List<Image> images;
@JsonIgnore
private Map<String, String> metadata;
@JsonIgnore
private AppStateService appStateService;
}
@@ -1,34 +0,0 @@
package tn.mnlr.vripper.event;
import lombok.Getter;
@Getter
public class Event<T> {
private final T data;
private final Kind kind;
private Event(Kind kind, T data) {
this.data = data;
this.kind = kind;
}
public static <T> Event<T> wrap(Kind kind, T data) {
return new Event<>(kind, data);
}
public enum Kind {
POST_UPDATE,
POST_REMOVE,
IMAGE_UPDATE,
METADATA_UPDATE,
QUEUED_UPDATE,
QUEUED_REMOVE,
LOG_EVENT_UPDATE,
LOG_EVENT_REMOVE,
VG_USER,
GLOBAL_STATE,
BYTES_PER_SECOND,
SETTINGS_UPDATE
}
}
@@ -1,27 +0,0 @@
package tn.mnlr.vripper.event;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Sinks;
import javax.annotation.PreDestroy;
@Service
public class EventBus {
public static final Sinks.EmitFailureHandler RETRY = (signalType, emitResult) -> true;
private final Sinks.Many<Event<?>> sink = Sinks.many().multicast().onBackpressureBuffer();
public void publishEvent(Event<?> event) {
sink.emitNext(event, RETRY);
}
public Flux<Event<?>> flux() {
return sink.asFlux();
}
@PreDestroy
private void destroy() {
sink.emitComplete(RETRY);
}
}
@@ -2,11 +2,11 @@ package tn.mnlr.vripper.exception;
public class DownloadException extends Exception {
public DownloadException(String message) {
super(message);
}
public DownloadException(String message) {
super(message);
}
public DownloadException(Throwable e) {
super(e);
}
public DownloadException(Exception e) {
super(e);
}
}
@@ -1,15 +1,15 @@
package tn.mnlr.vripper.exception;
public class HostException extends Exception {
public HostException(Throwable e) {
super(e);
}
public HostException(Exception e) {
super(e);
}
public HostException(String message) {
super(message);
}
public HostException(String message) {
super(message);
}
public HostException(String message, Throwable e) {
super(message, e);
}
public HostException(String message, Exception e) {
super(message,e);
}
}
@@ -2,7 +2,7 @@ package tn.mnlr.vripper.exception;
public class HtmlProcessorException extends Exception {
public HtmlProcessorException(Throwable e) {
super(e);
}
public HtmlProcessorException(Exception e) {
super(e);
}
}
@@ -1,16 +1,7 @@
package tn.mnlr.vripper.exception;
public class PostParseException extends Exception {
public PostParseException(String message) {
super(message);
}
public PostParseException(String message, Throwable e) {
super(message, e);
}
public PostParseException(Throwable e) {
super(e);
}
public PostParseException(Exception e) {
super(e);
}
}
@@ -1,16 +0,0 @@
package tn.mnlr.vripper.exception;
public class QueueException extends Exception {
public QueueException(String message) {
super(message);
}
public QueueException(String message, Throwable e) {
super(message, e);
}
public QueueException(Throwable e) {
super(e);
}
}
@@ -1,10 +0,0 @@
package tn.mnlr.vripper.exception;
import java.io.IOException;
public class RenameException extends Exception {
public RenameException(String message, IOException e) {
super(message, e);
}
}
@@ -2,7 +2,7 @@ package tn.mnlr.vripper.exception;
public class ValidationException extends Exception {
public ValidationException(String message) {
super(message);
}
public ValidationException(String message) {
super(message);
}
}
@@ -1,11 +1,11 @@
package tn.mnlr.vripper.exception;
public class VripperException extends Exception {
public VripperException(String message) {
super(message);
}
public VripperException(String message) {
super(message);
}
public VripperException(Throwable e) {
super(e);
}
public VripperException(Exception e) {
super(e);
}
}
@@ -1,7 +1,7 @@
package tn.mnlr.vripper.exception;
public class XpathException extends Exception {
public XpathException(Throwable e) {
super(e);
}
public XpathException(Exception e) {
super(e);
}
}
@@ -1,122 +1,100 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.ConnectionService;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.HtmlProcessorService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
import java.util.ArrayList;
import java.util.List;
@Service
@Slf4j
public class AcidimgHost extends Host {
private static final String host = "acidimg.cc";
private static final String CONTINUE_BUTTON_XPATH = "//input[@id='continuebutton']";
private static final String IMG_XPATH = "//img[@class='centred']";
private static final Logger logger = LoggerFactory.getLogger(AcidimgHost.class);
private final ConnectionService cm;
private final HostService hostService;
private final XpathService xpathService;
private final HtmlProcessorService htmlProcessorService;
private static final String host = "acidimg.cc";
private static final String CONTINUE_BUTTON_XPATH = "//input[@id='continuebutton']";
public static final String IMG_XPATH = "//img[@class='centred']";
@Autowired
public AcidimgHost(
ConnectionService cm,
HostService hostService,
XpathService xpathService,
HtmlProcessorService htmlProcessorService) {
this.cm = cm;
this.hostService = hostService;
this.xpathService = xpathService;
this.htmlProcessorService = htmlProcessorService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
Node contDiv;
try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
contDiv = xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH);
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
if (contDiv != null) {
log.debug(String.format("Click button found for %s", url));
HttpClient client = cm.getClient().build();
HttpPost httpPost = cm.buildHttpPost(url, context);
httpPost.addHeader("Referer", url);
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("imgContinue", "Continue to your image"));
try {
httpPost.setEntity(new UrlEncodedFormEntity(params));
} catch (Exception e) {
throw new HostException(e);
}
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
log.debug(String.format("Requesting %s", httpPost));
try (CloseableHttpResponse response =
(CloseableHttpResponse) client.execute(httpPost, context)) {
log.debug(String.format("Cleaning response for %s", httpPost));
doc = htmlProcessorService.clean(EntityUtils.toString(response.getEntity()));
EntityUtils.consumeQuietly(response.getEntity());
} catch (Exception e) {
throw new HostException(e);
}
Document doc = getResponse(url).getDocument();
Node contDiv;
try {
logger.info(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
contDiv = xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if (contDiv != null) {
logger.info(String.format("Click button found for %s", url));
HttpClient client = cm.getClient().build();
HttpPost httpPost = cm.buildHttpPost(url);
httpPost.addHeader("Referer", url);
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("imgContinue", "Continue to your image"));
try {
httpPost.setEntity(new UrlEncodedFormEntity(params));
} catch (Exception e) {
throw new HostException(e);
}
logger.info(String.format("Requesting %s", httpPost));
try (CloseableHttpResponse response = (CloseableHttpResponse) client.execute(httpPost)) {
logger.info(String.format("Cleaning response for %s", httpPost));
doc = htmlProcessorService.clean(EntityUtils.toString(response.getEntity()));
EntityUtils.consumeQuietly(response.getEntity());
} catch (Exception e) {
throw new HostException(e);
}
}
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if(imgNode == null) {
throw new HostException("Cannot find the image node");
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle.isEmpty() ? getDefaultImageName(imgUrl) : imgTitle);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if (imgNode == null) {
throw new HostException("Cannot find the image node");
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
return new HostService.NameUrl(
imgTitle.isEmpty() ? hostService.getDefaultImageName(imgUrl) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,43 +1,193 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import lombok.Getter;
import org.apache.http.Header;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.exception.DownloadException;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.exception.HtmlProcessorException;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.*;
import java.util.Objects;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Random;
@Service
@Slf4j
public abstract class Host {
abstract public class Host {
public abstract String getHost();
private static final Logger logger = LoggerFactory.getLogger(Host.class);
public abstract String getLookup();
private static final int READ_BUFFER_SIZE = 8192;
public boolean isSupported(String url) {
return url.contains(getLookup());
}
@Autowired
protected HtmlProcessorService htmlProcessorService;
public abstract HostService.NameUrl getNameAndUrl(
final String url, final HttpClientContext context) throws HostException;
@Autowired
protected XpathService xpathService;
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Host host = (Host) o;
return Objects.equals(getHost(), host.getHost());
}
@Autowired
private AppSettingsService appSettingsService;
@Override
public int hashCode() {
return Objects.hash(getHost());
}
@Autowired
private ConnectionManager cm;
@Override
public String toString() {
return getHost();
}
@Autowired
private DownloadSpeedService downloadSpeedService;
abstract protected String getHost();
public boolean isSupported(String url) {
return url.contains(getHost());
}
public void download(Image image, ImageFileData imageFileData) throws DownloadException, InterruptedException {
try {
imageFileData.setPageUrl(image.getUrl());
/**
* HOST SPECIFIC
*/
logger.info(String.format("Getting image url and name from %s using %s", image.getUrl(), this.getHost()));
setNameAndUrl(image.getUrl(), imageFileData);
logger.info(String.format("Resolved name for %s: %s", image.getUrl(), imageFileData.getImageName()));
logger.info(String.format("Resolved image url for %s: %s", image.getUrl(), imageFileData.getImageUrl()));
logger.info(String.format("Building image request for %s", image.getUrl()));
setImageRequest(imageFileData);
/**
* END HOST SPECIFIC
*/
if (!imageFileData.getImageName().toLowerCase().endsWith(".jpg") && !imageFileData.getImageName().toLowerCase().endsWith(".jpeg")) {
imageFileData.setImageName(imageFileData.getImageName() + ".jpg");
}
File destinationFolder = new File(appSettingsService.getDownloadPath(), sanitize(image.getPostName() + "_" + image.getPostId()));
logger.info(String.format("Saving to %s", destinationFolder.getPath()));
if (!destinationFolder.exists()) {
logger.info(String.format("Creating %s", destinationFolder.getPath()));
destinationFolder.mkdirs();
}
HttpClient client = cm.getClient().build();
logger.info(String.format("Downloading %s", imageFileData.getImageUrl()));
try (CloseableHttpResponse response = (CloseableHttpResponse) client.execute(imageFileData.getImageRequest())) {
if(response.getStatusLine().getStatusCode() / 100 != 2) {
EntityUtils.consumeQuietly(response.getEntity());
throw new DownloadException(String.format("Server returned code %d", response.getStatusLine().getStatusCode()));
}
try (
InputStream downloadStream = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream(sanitize(destinationFolder.getPath() + File.separator + imageFileData.getImageName()))
) {
image.setTotal(response.getEntity().getContentLength());
logger.info(String.format("%s length is %d", imageFileData.getImageUrl(), image.getTotal()));
logger.info(String.format("Starting data transfer for %s", imageFileData.getImageUrl()));
byte[] buffer = new byte[READ_BUFFER_SIZE];
int read;
while ((read = downloadStream.read(buffer, 0, READ_BUFFER_SIZE)) != -1) {
fos.write(buffer, 0, read);
image.increase(read);
downloadSpeedService.increase(read);
}
EntityUtils.consumeQuietly(response.getEntity());
}
}
} catch (Exception e) {
if(Thread.interrupted()) {
throw new InterruptedException("Download was interrupted");
}
throw new DownloadException(e);
}
}
/**
* Just for testing, you may ignore
* @throws Exception
*/
private void randomFail() throws Exception {
Random random = new Random();
int i = random.nextInt(100);
if (i < 50) {
throw new Exception("Purpose error");
}
}
protected final String sanitize(final String folderName) {
String sanitizedFolderName = folderName.replaceAll("\\.|\\\\|/|\\||:|\\?|\\*|\"|<|>|\\p{Cntrl}", "_");
logger.debug(String.format("%s sanitized to %s", folderName, sanitizedFolderName));
return sanitizedFolderName;
}
protected final String getDefaultImageName(final String imgUrl) {
String imageTitle = imgUrl.substring(imgUrl.lastIndexOf('/') + 1);
logger.debug(String.format("Extracting name from url %s: %s", imgUrl, imageTitle));
return imgUrl;
}
protected final Response getResponse(final String url) throws HostException {
String basePage;
HttpClient client = cm.getClient().build();
HttpGet httpGet = cm.buildHttpGet(url);
Header[] headers;
logger.info(String.format("Requesting %s", url));
try (CloseableHttpResponse response = (CloseableHttpResponse) client.execute(httpGet)) {
if (response.getStatusLine().getStatusCode() / 100 != 2) {
throw new HostException(String.format("Unexpected response code: %d", response.getStatusLine().getStatusCode()));
}
headers = response.getAllHeaders();
basePage = EntityUtils.toString(response.getEntity());
logger.debug(String.format("%s response: %n%s", url, basePage));
EntityUtils.consumeQuietly(response.getEntity());
} catch (IOException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Cleaning %s response", url));
return new Response(htmlProcessorService.clean(basePage), headers);
} catch (HtmlProcessorException e) {
throw new HostException(e);
}
}
protected void setImageRequest(final ImageFileData imageFileData) {
HttpGet httpGet = cm.buildHttpGet(imageFileData.getImageUrl());
httpGet.addHeader("Referer", imageFileData.getPageUrl());
imageFileData.setImageRequest(httpGet);
}
protected abstract void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException;
@Getter
public static class Response {
protected Response(Document document, Header[] headers) {
this.document = document;
this.headers = headers;
}
private Document document;
private Header[] headers;
}
}
@@ -1,96 +1,99 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.Header;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.HtmlProcessorException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
import java.util.Optional;
import java.util.UUID;
import java.io.IOException;
import java.util.Arrays;
import java.util.stream.Collectors;
@Service
@Slf4j
public class ImageBamHost extends Host {
private static final String host = "imagebam.com";
private static final String IMG_XPATH = "//img[contains(@class,'main-image')]";
private static final String CONTINUE_XPATH = "//*[contains(text(), 'Continue')]";
private static final Logger logger = LoggerFactory.getLogger(ImageBamHost.class);
private final HostService hostService;
private final XpathService xpathService;
private static final String host = "imagebam.com";
public static final String CONTINUE_BUTTON_XPATH = "//a[@title='Continue to your image']";
public static final String IMG_XPATH = "//img[@class='image']";
@Autowired
public ImageBamHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
HostService.Response response = hostService.getResponse(url, context);
Document doc = response.getDocument();
try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_XPATH, url));
if (xpathService.getAsNode(doc, CONTINUE_XPATH) != null) {
// Button detected. No need to actually click it, just make the call again.
response = hostService.getResponse(url, context);
doc = response.getDocument();
}
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
Response response = getResponse(url);
Document doc = response.getDocument();
Header[] headers = response.getHeaders();
String cookies = Arrays.asList(headers)
.stream()
.filter(e -> e.getName().toLowerCase().contains("Set-Cookie".toLowerCase()))
.map(e -> e.getValue().split(";")[0].trim())
.collect(Collectors.joining("; "));
Node contDiv;
try {
logger.info(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
contDiv = xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if (contDiv != null) {
logger.info(String.format("Getting cookies to use for %s", url));
logger.info(String.format("Click button found for %s", url));
HttpClient client = cm.getClient().build();
HttpGet httpGet = cm.buildHttpGet(url);
httpGet.addHeader("Referer", url);
httpGet.addHeader("Cookie", cookies);
logger.info(String.format("Requesting %s", httpGet));
try (CloseableHttpResponse res = (CloseableHttpResponse) client.execute(httpGet)) {
String s = EntityUtils.toString(res.getEntity());
logger.debug(String.format("%s response is:%n%s", httpGet, s));
logger.debug(String.format("Cleaning response for %s", httpGet));
doc = htmlProcessorService.clean(s);
EntityUtils.consumeQuietly(res.getEntity());
} catch (IOException | HtmlProcessorException e) {
throw new HostException(e);
}
}
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("id").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle);
} catch(Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
if (imgNode == null) {
throw new HostException(String.format("Xpath '%s' cannot be found in '%s'", IMG_XPATH, url));
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle =
Optional.ofNullable(imgNode.getAttributes().getNamedItem("alt"))
.map(e -> e.getTextContent().trim())
.orElse("");
String imgUrl =
Optional.ofNullable(imgNode.getAttributes().getNamedItem("src"))
.map(e -> e.getTextContent().trim())
.orElse("");
String defaultName = UUID.randomUUID().toString();
int index = imgUrl.lastIndexOf('/');
if (index != -1 && index < imgUrl.length()) {
defaultName = imgUrl.substring(imgUrl.lastIndexOf('/') + 1);
}
return new HostService.NameUrl(imgTitle.isEmpty() ? defaultName : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,70 +1,54 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
import java.util.Optional;
@Service
@Slf4j
public class ImageTwistHost extends Host {
private static final String IMG_XPATH = "//img[contains(@class, 'img')]";
private static final String host = "imagetwist.com";
public static final String IMG_XPATH = "//img[contains(@class, 'img')]";
private static final Logger logger = LoggerFactory.getLogger(ImageTwistHost.class);
private static final String host = "imagetwist.com";
@Autowired
private ConnectionManager cm;
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public ImageTwistHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle =
Optional.ofNullable(imgNode.getAttributes().getNamedItem("alt"))
.map(Node::getTextContent)
.map(String::trim)
.orElse(null);
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
return new HostService.NameUrl(imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
Document doc = getResponse(url).getDocument();
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = Optional.ofNullable(imgNode.getAttributes().getNamedItem("alt")).map(Node::getTextContent).map(String::trim).orElse(null);
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,84 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
@Service
@Slf4j
public class ImageVenueHost extends Host {
private static final String host = "imagevenue.com";
private static final String CONTINUE_BUTTON_XPATH = "//a[@title='Continue to ImageVenue']";
private static final String IMG_XPATH = "//a[@data-toggle='full']/img";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public ImageVenueHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
HostService.Response resp = hostService.getResponse(url, context);
Document doc = resp.getDocument();
try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
if (xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH) != null) {
// Button detected. No need to actually click it, just make the call again.
resp = hostService.getResponse(url, context);
doc = resp.getDocument();
}
} catch (XpathException e) {
throw new HostException(e);
}
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if (imgNode == null) {
throw new HostException("Failed to locate image");
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
return new HostService.NameUrl(
imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,71 +1,60 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
@Service
@Slf4j
public class ImageZillaHost extends Host {
private static final String host = "imagezilla.net";
private static final String lookup = "imagezilla.net/show";
private static final String IMG_XPATH = "//img[@id='photo']";
private static final Logger logger = LoggerFactory.getLogger(ImageZillaHost.class);
private final HostService hostService;
private final XpathService xpathService;
private static final String host = "imagezilla.net";
public static final String IMG_XPATH = "//img[@id='photo']";
@Autowired
public ImageZillaHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return lookup;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
String title;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
Node titleNode = xpathService.getAsNode(doc, IMG_XPATH).getAttributes().getNamedItem("title");
log.debug(String.format("Resolving name for %s", url));
if (titleNode != null) {
title = titleNode.getTextContent().trim();
} else {
title = null;
}
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
if (title == null || title.isEmpty()) {
title = hostService.getDefaultImageName(url);
}
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
try {
return new HostService.NameUrl(title, url.replace("show", "images"));
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
Document doc = getResponse(url).getDocument();
String title;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
Node titleNode = xpathService.getAsNode(doc, IMG_XPATH).getAttributes().getNamedItem("title");
logger.info(String.format("Resolving name for %s", url));
if(titleNode != null) {
title = titleNode.getTextContent().trim();
} else {
title = null;
}
} catch (XpathException e) {
throw new HostException(e);
}
if(title == null || title.isEmpty()) {
title = getDefaultImageName(url);
}
try {
imageFileData.setImageUrl(url.replace("show", "images"));
imageFileData.setImageName(title);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,67 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
@Service
@Slf4j
public class ImgSpiceHost extends Host {
private static final String host = "imgspice.com";
private static final String IMG_XPATH = "//img[@id='imgpreview']";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public ImgSpiceHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
HostService.Response resp = hostService.getResponse(url, context);
Document doc = resp.getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
return new HostService.NameUrl(
imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,64 +1,54 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
@Service
@Slf4j
public class ImgboxHost extends Host {
private static final String host = "imgbox.com";
private static final String IMG_XPATH = "//img[@id='img']";
private static final Logger logger = LoggerFactory.getLogger(ImgboxHost.class);
private final HostService hostService;
private final XpathService xpathService;
private static final String host = "imgbox.com";
public static final String IMG_XPATH = "//img[@id='img']";
@Autowired
public ImgboxHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("title").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
return new HostService.NameUrl(imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
Document doc = getResponse(url).getDocument();
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("title").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,14 +1,14 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
@@ -16,114 +16,82 @@ import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.HtmlProcessorException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.ConnectionService;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.HtmlProcessorService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Service
@Slf4j
public class ImxHost extends Host {
private static final String host = "imx.to";
private static final String CONTINUE_BUTTON_XPATH = "//*[@name='imgContinue']";
private static final String IMG_XPATH = "//img[@class='centred']";
private static final Logger logger = LoggerFactory.getLogger(ImxHost.class);
private final HostService hostService;
private final XpathService xpathService;
private final ConnectionService cm;
private final HtmlProcessorService htmlProcessorService;
private static final String host = "imx.to";
public static final String CONTINUE_BUTTON_XPATH = "//div[@id='continuetoimage']";
public static final String IMG_XPATH = "//img[@class='centred']";
@Autowired
public ImxHost(
HostService hostService,
XpathService xpathService,
ConnectionService cm,
HtmlProcessorService htmlProcessorService) {
this.hostService = hostService;
this.xpathService = xpathService;
this.cm = cm;
this.htmlProcessorService = htmlProcessorService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
HostService.Response resp = hostService.getResponse(url, context);
Document doc = resp.getDocument();
Node contDiv;
String value = null;
try {
log.debug(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
contDiv = xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH);
if (contDiv == null) {
throw new HostException(CONTINUE_BUTTON_XPATH + " cannot be found");
}
Node node = contDiv.getAttributes().getNamedItem("value");
if (node != null) {
value = node.getTextContent();
}
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
if (value == null) {
throw new HostException("Failed to obtain value attribute from continue input");
}
log.debug(String.format("Click button found for %s", url));
HttpClient client = cm.getClient().build();
HttpPost httpPost = cm.buildHttpPost(url, context);
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("imgContinue", value));
try {
httpPost.setEntity(new UrlEncodedFormEntity(params));
} catch (Exception e) {
throw new HostException(e);
}
log.debug(String.format("Requesting %s", httpPost));
try (CloseableHttpResponse response =
(CloseableHttpResponse) client.execute(httpPost, context)) {
log.debug(String.format("Cleaning response for %s", httpPost));
doc = htmlProcessorService.clean(EntityUtils.toString(response.getEntity()));
EntityUtils.consumeQuietly(response.getEntity());
} catch (IOException | HtmlProcessorException e) {
throw new HostException(e);
}
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
Document doc = getResponse(url).getDocument();
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
Node contDiv;
try {
logger.info(String.format("Looking for xpath expression %s in %s", CONTINUE_BUTTON_XPATH, url));
contDiv = xpathService.getAsNode(doc, CONTINUE_BUTTON_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
return new HostService.NameUrl(
imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
if (contDiv != null) {
logger.info(String.format("Click button found for %s", url));
HttpClient client = cm.getClient().build();
HttpPost httpPost = cm.buildHttpPost(url);
httpPost.addHeader("Referer", url);
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("imgContinue", "Continue to image ... "));
try {
httpPost.setEntity(new UrlEncodedFormEntity(params));
} catch (Exception e) {
throw new HostException(e);
}
logger.info(String.format("Requesting %s", httpPost));
try (CloseableHttpResponse response = (CloseableHttpResponse) client.execute(httpPost)) {
logger.debug(String.format("Cleaning response for %s", httpPost));
doc = htmlProcessorService.clean(EntityUtils.toString(response.getEntity()));
EntityUtils.consumeQuietly(response.getEntity());
} catch (IOException | HtmlProcessorException e) {
throw new HostException(e);
}
}
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle);
} catch(Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,75 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
@Service
@Slf4j
public class PimpandhostHost extends Host {
private static final String host = "pimpandhost.com";
private static final String IMG_XPATH = "//img[contains(@class, 'original')]";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public PimpandhostHost(HostService hostService, XpathService xpathService) {
super();
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url;
try {
url = hostService.appendUri(_url.replace("http://", "https://"), "size=original");
} catch (Exception e) {
throw new HostException(e);
}
HostService.Response resp = hostService.getResponse(url, context);
Document doc = resp.getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl =
"https:" + imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
return new HostService.NameUrl(
imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,69 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
@Service
@Slf4j
public class PixRouteHost extends Host {
private static final String host = "pixroute.com";
private static final String IMG_XPATH = "//img[@id='imgpreview']";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public PixRouteHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
Document doc = hostService.getResponse(url, context).getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
if (imgNode == null) {
throw new HostException("Failed to locate image");
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
return new HostService.NameUrl(
imgNode.getAttributes().getNamedItem("alt").getTextContent().trim(),
imgNode.getAttributes().getNamedItem("src").getTextContent().trim());
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,65 +1,54 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
@Service
@Slf4j
public class PixhostHost extends Host {
private static final String host = "pixhost.to";
private static final String lookup = "pixhost.to/show";
private static final String IMG_XPATH = "//img[@id='image']";
private static final Logger logger = LoggerFactory.getLogger(PixhostHost.class);
private final HostService hostService;
private final XpathService xpathService;
private static final String host = "pixhost.to";
public static final String IMG_XPATH = "//img[@id='image']";
@Autowired
public PixhostHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return lookup;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
return new HostService.NameUrl(imgTitle.substring(imgTitle.indexOf('_') + 1), imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
Document doc = getResponse(url).getDocument();
Node imgNode;
try {
logger.info(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
logger.info(String.format("Resolving name and image url for %s", url));
String imgTitle = imgNode.getAttributes().getNamedItem("alt").getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
imageFileData.setImageUrl(imgUrl);
imageFileData.setImageName(imgTitle.substring(imgTitle.indexOf('_') + 1));
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,72 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
@Service
@Slf4j
public class PixxxelsHost extends Host {
private static final String host = "pixxxels.cc";
private static final String IMG_XPATH = "//*[@id='download']";
private static final String TITLE_XPATH = "//*[contains(@class,'imagename')]";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public PixxxelsHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
HostService.Response resp = hostService.getResponse(url, context);
Document doc = resp.getDocument();
Node imgNode, titleNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url));
titleNode = xpathService.getAsNode(doc, TITLE_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle = titleNode.getTextContent().trim();
String imgUrl = imgNode.getAttributes().getNamedItem("href").getTextContent().trim();
return new HostService.NameUrl(
imgTitle.isEmpty() ? imgUrl.substring(imgUrl.lastIndexOf('/') + 1) : imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,74 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import java.util.Optional;
@Service
@Slf4j
public class PostImgHost extends Host {
private static final String host = "postimg.cc";
private static final String TITLE_XPATH = "//span[contains(@class,'imagename')]";
private static final String IMG_XPATH = "//a[@id='download']";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public PostImgHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String _url, final HttpClientContext context)
throws HostException {
String url = _url.replace("http://", "https://");
Document doc = hostService.getResponse(url, context).getDocument();
Node urlNode, titleNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url));
titleNode = xpathService.getAsNode(doc, TITLE_XPATH);
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
urlNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle =
Optional.ofNullable(titleNode)
.map(node -> node.getTextContent().trim())
.orElseGet(() -> hostService.getDefaultImageName(url));
return new HostService.NameUrl(
imgTitle, urlNode.getAttributes().getNamedItem("href").getTextContent().trim());
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,73 +1,62 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import tn.mnlr.vripper.q.ImageFileData;
import tn.mnlr.vripper.services.ConnectionManager;
@Service
@Slf4j
public class TurboImageHost extends Host {
private static final String host = "turboimagehost.com";
private static final String TITLE_XPATH = "//div[contains(@class,'titleFullS')]/h1";
private static final String IMG_XPATH = "//img[@id='imageid']";
private static final Logger logger = LoggerFactory.getLogger(TurboImageHost.class);
private final HostService hostService;
private final XpathService xpathService;
private static final String host = "turboimagehost.com";
public static final String TITLE_XPATH = "//div[contains(@class,'titleFullS')]/h1";
public static final String IMG_XPATH = "//img[@id='uImage']";
@Autowired
public TurboImageHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Autowired
private ConnectionManager cm;
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
String title;
try {
log.debug(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url));
Node titleNode = xpathService.getAsNode(doc, TITLE_XPATH);
log.debug(String.format("Resolving name for %s", url));
if (titleNode != null) {
title = titleNode.getTextContent().trim();
} else {
title = null;
}
} catch (XpathException e) {
throw new HostException(e);
@Override
protected String getHost() {
return host;
}
if (title == null || title.isEmpty()) {
title = hostService.getDefaultImageName(url);
}
@Override
protected void setNameAndUrl(final String url, final ImageFileData imageFileData) throws HostException {
try {
Node urlNode = xpathService.getAsNode(doc, IMG_XPATH);
return new HostService.NameUrl(
title, urlNode.getAttributes().getNamedItem("src").getTextContent().trim());
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
Document doc = getResponse(url).getDocument();
String title;
try {
logger.info(String.format("Looking for xpath expression %s in %s", TITLE_XPATH, url));
Node titleNode = xpathService.getAsNode(doc, TITLE_XPATH);
logger.info(String.format("Resolving name for %s", url));
if(titleNode != null) {
title = titleNode.getTextContent().trim();
} else {
title = null;
}
} catch (XpathException e) {
throw new HostException(e);
}
if(title == null || title.isEmpty()) {
title = getDefaultImageName(url);
}
try {
Node urlNode = xpathService.getAsNode(doc, IMG_XPATH);
imageFileData.setImageUrl(urlNode.getAttributes().getNamedItem("src").getTextContent().trim());
imageFileData.setImageName(title);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
}
@@ -1,70 +0,0 @@
package tn.mnlr.vripper.host;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.protocol.HttpClientContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.XpathException;
import tn.mnlr.vripper.services.HostService;
import tn.mnlr.vripper.services.XpathService;
import java.util.Optional;
@Service
@Slf4j
public class ViprImHost extends Host {
private static final String IMG_XPATH = "//img[contains(@class, 'img')]";
private static final String host = "vipr.im";
private final HostService hostService;
private final XpathService xpathService;
@Autowired
public ViprImHost(HostService hostService, XpathService xpathService) {
this.hostService = hostService;
this.xpathService = xpathService;
}
@Override
public String getHost() {
return host;
}
@Override
public String getLookup() {
return host;
}
@Override
public HostService.NameUrl getNameAndUrl(final String url, final HttpClientContext context)
throws HostException {
Document doc = hostService.getResponse(url, context).getDocument();
Node imgNode;
try {
log.debug(String.format("Looking for xpath expression %s in %s", IMG_XPATH, url));
imgNode = xpathService.getAsNode(doc, IMG_XPATH);
} catch (XpathException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Resolving name and image url for %s", url));
String imgTitle =
Optional.ofNullable(imgNode.getAttributes().getNamedItem("alt"))
.map(Node::getTextContent)
.map(String::trim)
.orElse(null);
String imgUrl = imgNode.getAttributes().getNamedItem("src").getTextContent().trim();
return new HostService.NameUrl(imgTitle, imgUrl);
} catch (Exception e) {
throw new HostException("Unexpected error occurred", e);
}
}
}
@@ -1,79 +0,0 @@
package tn.mnlr.vripper.jpa;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import tn.mnlr.vripper.download.DownloadService;
import tn.mnlr.vripper.services.ThreadPoolService;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.io.File;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Component
@EnableScheduling
public class Management {
public static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");
private static final Pattern BACKUP_FILE_PATTERN =
Pattern.compile("^db_(\\d{4}-\\d{2}-\\d{2})\\.tar\\.gz$");
private final JdbcTemplate jdbcTemplate;
private final String backupFolder;
private final ThreadPoolService threadPoolService;
private final DownloadService downloadService;
public Management(
JdbcTemplate jdbcTemplate,
@Value("${base.dir}") String baseDir,
@Value("${base.dir.name}") String baseDirName,
ThreadPoolService threadPoolService,
DownloadService downloadService) {
this.jdbcTemplate = jdbcTemplate;
this.threadPoolService = threadPoolService;
this.downloadService = downloadService;
this.backupFolder = baseDir + File.separator + baseDirName + File.separator + "backup";
}
@PreDestroy
public void destroy() {
try {
threadPoolService.destroy();
downloadService.destroy();
this.jdbcTemplate.execute("SHUTDOWN");
} catch (Exception ignored) {
Thread.currentThread().interrupt();
}
}
@PostConstruct
@Scheduled(cron = "0 0 0 ? * *")
private void backup() {
for (File file :
Optional.ofNullable(new File(backupFolder).listFiles()).orElse(new File[] {})) {
Matcher matcher = BACKUP_FILE_PATTERN.matcher(file.getName());
if (matcher.find()) {
LocalDate localDate = LocalDate.parse(matcher.group(1), FORMATTER);
if (localDate.isEqual(LocalDate.now())) {
return;
}
}
}
// create a backup file
File backupFile =
new File(
backupFolder, String.format("db_%s.tar.gz", LocalDateTime.now().format(FORMATTER)));
jdbcTemplate.execute(String.format("BACKUP DATABASE TO '%s' BLOCKING", backupFile.getPath()));
}
}
@@ -1,25 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.ser.std.StdSerializer;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
public class DateTimeSerializer extends StdSerializer<LocalDateTime> {
private static final DateTimeFormatter DATE_TIME_FORMATTER =
DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss.SSS");
protected DateTimeSerializer() {
super(LocalDateTime.class);
}
@Override
public void serialize(LocalDateTime value, JsonGenerator gen, SerializerProvider provider)
throws IOException {
gen.writeString(value.format(DATE_TIME_FORMATTER));
}
}
@@ -1,66 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import com.fasterxml.jackson.annotation.JsonIgnore;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import java.util.Objects;
@Getter
@Setter
@ToString
@NoArgsConstructor
public class Image {
@JsonIgnore protected Long id;
@JsonIgnore private Host host;
private String url;
private int index;
private long current = 0;
private long total = 0;
private Status status;
private String postId;
@JsonIgnore private Long postIdRef;
public Image(String postId, String url, Host host, int index) {
this.postId = postId;
this.url = url;
this.host = host;
this.index = index;
status = Status.STOPPED;
}
public void increase(int read) {
current += read;
}
public void init() {
current = 0;
status = Status.STOPPED;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Image image = (Image) o;
return Objects.equals(url, image.url);
}
@Override
public int hashCode() {
return Objects.hash(url);
}
}
@@ -1,52 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import java.time.LocalDateTime;
@Getter
@Setter
@ToString
@NoArgsConstructor
public class LogEvent {
private Long id;
private Type type;
private Status status;
@JsonSerialize(using = DateTimeSerializer.class)
private LocalDateTime time;
private String message;
public LogEvent(Type type, Status status, LocalDateTime time, String message) {
this.type = type;
this.status = status;
this.time = time;
this.message = message;
}
public enum Type {
POST,
QUEUED,
THANKS,
METADATA,
SCAN,
DOWNLOAD,
// Below are deprecated events
METADATA_CACHE_MISS,
QUEUED_CACHE_MISS
}
public enum Status {
PENDING,
PROCESSING,
DONE,
ERROR
}
}
@@ -1,31 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import java.util.Collections;
import java.util.List;
@Getter
@Setter
@NoArgsConstructor
public class Metadata {
private Long postIdRef;
private String postId;
private String postedBy;
private List<String> resolvedNames = Collections.emptyList();
public static Metadata from(Metadata metadata) {
Metadata copy = new Metadata();
copy.postIdRef = metadata.postIdRef;
copy.postId = metadata.postId;
copy.postedBy = metadata.postedBy;
copy.resolvedNames = List.copyOf(metadata.resolvedNames);
return copy;
}
}
@@ -1,90 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import java.time.LocalDateTime;
import java.util.Collections;
import java.util.Objects;
import java.util.Set;
@Getter
@Setter
@ToString
@NoArgsConstructor
public class Post {
@JsonIgnore private Long id;
private Status status;
private String postId;
private String threadTitle;
private String threadId;
private String title;
private String url;
private int done;
private int total;
private Set<String> hosts;
private String forum;
@JsonIgnore private String securityToken;
@JsonIgnore private String downloadDirectory;
private boolean thanked;
private Set<String> previews = Collections.emptySet();
private Metadata metadata;
private int rank = Integer.MAX_VALUE;
@JsonSerialize(using = DateTimeSerializer.class)
private LocalDateTime addedOn;
public Post(
String title,
String url,
String postId,
String threadId,
String threadTitle,
String forum,
String securityToken) {
this.title = title;
this.url = url;
this.postId = postId;
this.forum = forum;
this.threadId = threadId;
this.threadTitle = threadTitle;
this.securityToken = securityToken;
status = Status.STOPPED;
addedOn = LocalDateTime.now();
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Post post = (Post) o;
return Objects.equals(postId, post.postId);
}
@Override
public int hashCode() {
return Objects.hash(postId);
}
}
@@ -1,56 +0,0 @@
package tn.mnlr.vripper.jpa.domain;
import com.fasterxml.jackson.annotation.JsonIgnore;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import java.util.Objects;
@Getter
@Setter
@NoArgsConstructor
@ToString
public class Queued {
@JsonIgnore private Long id;
private String link;
private String threadId;
private String postId;
private int total = 0;
private boolean loading = true;
public Queued(String link, String threadId, String postId) {
this();
this.link = link;
this.threadId = threadId;
this.postId = postId;
}
public void done() {
this.loading = false;
}
public void increment() {
this.total++;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Queued that = (Queued) o;
return Objects.equals(threadId, that.threadId);
}
@Override
public int hashCode() {
return Objects.hash(threadId);
}
}
@@ -1,10 +0,0 @@
package tn.mnlr.vripper.jpa.domain.enums;
public enum Status {
PENDING,
DOWNLOADING,
COMPLETE,
ERROR,
STOPPED,
PARTIAL
}
@@ -1,32 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
import tn.mnlr.vripper.jpa.domain.Image;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import java.util.List;
import java.util.Optional;
public interface IImageRepository extends IRepository {
Image save(Image image);
void deleteAllByPostId(String postId);
List<Image> findByPostId(String postId);
Integer countError();
List<Image> findByPostIdAndIsNotCompleted(String postId);
int stopByPostIdAndIsNotCompleted(String postId);
List<Image> findByPostIdAndIsError(String postId);
Optional<Image> findById(Long id);
int updateStatus(Status status, Long id);
int updateCurrent(long current, Long id);
int updateTotal(long total, Long id);
}
@@ -1,21 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
import tn.mnlr.vripper.jpa.domain.LogEvent;
import java.util.List;
import java.util.Optional;
public interface ILogEventRepository extends IRepository {
LogEvent save(LogEvent logEvent);
LogEvent update(LogEvent logEvent);
List<LogEvent> findAll();
Optional<LogEvent> findById(Long id);
void delete(Long id);
void deleteAll();
}
@@ -1,14 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
import tn.mnlr.vripper.jpa.domain.Metadata;
import java.util.Optional;
public interface IMetadataRepository extends IRepository {
Metadata save(Metadata metadata);
Optional<Metadata> findByPostId(String postId);
int deleteByPostId(String postId);
}
@@ -1,38 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
import tn.mnlr.vripper.jpa.domain.Post;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import java.util.List;
import java.util.Optional;
public interface IPostRepository extends IRepository {
Post save(Post post);
Optional<Post> findByPostId(String postId);
Optional<Post> findById(Long id);
List<String> findCompleted();
List<Post> findAll();
boolean existByPostId(String postId);
int setDownloadingToStopped();
int deleteByPostId(String postId);
int updateStatus(Status status, Long id);
int updateDone(int done, Long id);
int updateFolderName(String postFolderName, Long id);
int updateTitle(String title, Long id);
int updateThanked(boolean thanked, Long id);
int updateRank(int rank, Long id);
}
@@ -1,20 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
import tn.mnlr.vripper.jpa.domain.Queued;
import java.util.List;
import java.util.Optional;
public interface IQueuedRepository extends IRepository {
Queued save(Queued queued);
Optional<Queued> findByThreadId(String threadId);
List<Queued> findAll();
Optional<Queued> findById(Long id);
int deleteByThreadId(String threadId);
void deleteAll();
}
@@ -1,3 +0,0 @@
package tn.mnlr.vripper.jpa.repositories;
public interface IRepository {}
@@ -1,156 +0,0 @@
package tn.mnlr.vripper.jpa.repositories.impl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.SpringContext;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.jpa.domain.Image;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.jpa.repositories.IImageRepository;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.List;
import java.util.Optional;
@Service
public class ImageRepository implements IImageRepository {
private final JdbcTemplate jdbcTemplate;
private final EventBus eventBus;
@Autowired
public ImageRepository(JdbcTemplate jdbcTemplate, EventBus eventBus) {
this.jdbcTemplate = jdbcTemplate;
this.eventBus = eventBus;
}
private synchronized Long nextId() {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_IMAGE", Long.class);
}
@Override
public Image save(Image image) {
long id = nextId();
jdbcTemplate.update(
"INSERT INTO IMAGE (ID, CURRENT, HOST, INDEX, POST_ID, STATUS, TOTAL, URL, POST_ID_REF) VALUES (?,?,?,?,?,?,?,?,?)",
id,
image.getCurrent(),
image.getHost().getHost(),
image.getIndex(),
image.getPostId(),
image.getStatus().name(),
image.getTotal(),
image.getUrl(),
image.getPostIdRef());
image.setId(id);
eventBus.publishEvent(Event.wrap(Event.Kind.IMAGE_UPDATE, id));
return image;
}
@Override
public void deleteAllByPostId(String postId) {
jdbcTemplate.update("DELETE FROM IMAGE WHERE POST_ID = ?", postId);
}
@Override
public List<Image> findByPostId(String postId) {
return jdbcTemplate.query(
"SELECT * FROM IMAGE WHERE POST_ID = ?", new ImageRowMapper(), postId);
}
@Override
public Integer countError() {
return jdbcTemplate.queryForObject(
"SELECT COUNT(*) FROM IMAGE AS image WHERE image.STATUS = 'ERROR'", Integer.class);
}
@Override
public List<Image> findByPostIdAndIsNotCompleted(String postId) {
return jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.POST_ID = ? AND image.STATUS <> 'COMPLETE'",
new ImageRowMapper(),
postId);
}
@Override
public int stopByPostIdAndIsNotCompleted(String postId) {
return jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.STATUS = 'STOPPED' WHERE image.POST_ID = ? AND image.STATUS <> 'COMPLETE'",
postId);
}
@Override
public List<Image> findByPostIdAndIsError(String postId) {
return jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.POST_ID = ? AND image.STATUS = 'ERROR'",
new ImageRowMapper(),
postId);
}
@Override
public Optional<Image> findById(Long id) {
List<Image> images =
jdbcTemplate.query(
"SELECT * FROM IMAGE AS image WHERE image.ID = ?", new ImageRowMapper(), id);
if (images.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(images.get(0));
}
}
@Override
public int updateStatus(Status status, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.STATUS = ? WHERE image.ID = ?", status.name(), id);
eventBus.publishEvent(Event.wrap(Event.Kind.IMAGE_UPDATE, id));
return mutationCount;
}
@Override
public int updateCurrent(long current, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.CURRENT = ? WHERE image.ID = ?", current, id);
eventBus.publishEvent(Event.wrap(Event.Kind.IMAGE_UPDATE, id));
return mutationCount;
}
@Override
public int updateTotal(long total, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE IMAGE AS image SET image.TOTAL = ? WHERE image.ID = ?", total, id);
eventBus.publishEvent(Event.wrap(Event.Kind.IMAGE_UPDATE, id));
return mutationCount;
}
}
class ImageRowMapper implements RowMapper<Image> {
@Override
public Image mapRow(ResultSet rs, int rowNum) throws SQLException {
Image image = new Image();
image.setId(rs.getLong("ID"));
String host = rs.getString("HOST");
image.setHost(
SpringContext.getBeansOfType(Host.class).values().stream()
.filter(e -> e.getHost().equals(host))
.findAny()
.orElse(null));
image.setUrl(rs.getString("URL"));
image.setIndex(rs.getInt("INDEX"));
image.setCurrent(rs.getLong("CURRENT"));
image.setTotal(rs.getLong("TOTAL"));
image.setStatus(Status.valueOf(rs.getString("STATUS")));
image.setPostId(rs.getString("POST_ID"));
image.setPostIdRef(rs.getLong("POST_ID_REF"));
return image;
}
}
@@ -1,125 +0,0 @@
package tn.mnlr.vripper.jpa.repositories.impl;
import lombok.extern.slf4j.Slf4j;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.lang.NonNull;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.jpa.domain.LogEvent;
import tn.mnlr.vripper.jpa.repositories.ILogEventRepository;
import tn.mnlr.vripper.services.SettingsService;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;
import java.util.Optional;
@Service
@Slf4j
public class LogEventRepository implements ILogEventRepository {
private final JdbcTemplate jdbcTemplate;
private final SettingsService settingsService;
private final EventBus eventBus;
public LogEventRepository(
JdbcTemplate jdbcTemplate, SettingsService settingsService, EventBus eventBus) {
this.jdbcTemplate = jdbcTemplate;
this.settingsService = settingsService;
this.eventBus = eventBus;
}
private synchronized Long nextId() {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_EVENT", Long.class);
}
@Override
public synchronized LogEvent save(@NonNull LogEvent logEvent) {
int maxRecords = settingsService.getSettings().getMaxEventLog() - 1;
Long count = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM EVENT", Long.class);
if (count > maxRecords) {
List<Long> idList =
jdbcTemplate.queryForList(
"SELECT ID FROM EVENT ORDER BY TIME ASC LIMIT ?", Long.class, count - maxRecords);
idList.forEach(this::delete);
}
long id = nextId();
jdbcTemplate.update(
"INSERT INTO EVENT (ID, TYPE, STATUS, TIME, MESSAGE) VALUES (?,?,?,?,?)",
id,
logEvent.getType().name(),
logEvent.getStatus().name(),
logEvent.getTime().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME),
logEvent.getMessage());
logEvent.setId(id);
eventBus.publishEvent(Event.wrap(Event.Kind.LOG_EVENT_UPDATE, id));
return logEvent;
}
@Override
public LogEvent update(@NonNull LogEvent logEvent) {
if (logEvent.getId() == null) {
log.warn("Cannot update entity with null id");
return logEvent;
}
jdbcTemplate.update(
"UPDATE EVENT SET TYPE = ?, STATUS = ?, TIME = ?, MESSAGE = ? WHERE ID = ?",
logEvent.getType().name(),
logEvent.getStatus().name(),
logEvent.getTime().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME),
logEvent.getMessage(),
logEvent.getId());
eventBus.publishEvent(Event.wrap(Event.Kind.LOG_EVENT_UPDATE, logEvent.getId()));
return logEvent;
}
@Override
public Optional<LogEvent> findById(Long id) {
List<LogEvent> logEvents =
jdbcTemplate.query("SELECT * FROM EVENT WHERE ID = ?", new LogEventRowMapper(), id);
if (logEvents.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(logEvents.get(0));
}
}
@Override
public List<LogEvent> findAll() {
return jdbcTemplate.query("SELECT * FROM EVENT", new LogEventRowMapper());
}
@Override
public void delete(Long id) {
jdbcTemplate.update("DELETE FROM EVENT WHERE ID = ?", id);
eventBus.publishEvent(Event.wrap(Event.Kind.LOG_EVENT_REMOVE, id));
}
@Override
public void deleteAll() {
jdbcTemplate.update("DELETE FROM EVENT");
}
}
class LogEventRowMapper implements RowMapper<LogEvent> {
@Override
public LogEvent mapRow(ResultSet rs, int rowNum) throws SQLException {
LogEvent logEvent = new LogEvent();
logEvent.setId(rs.getLong("ID"));
logEvent.setType(LogEvent.Type.valueOf(rs.getString("TYPE")));
logEvent.setStatus(LogEvent.Status.valueOf(rs.getString("STATUS")));
logEvent.setTime(
LocalDateTime.parse(rs.getString("TIME"), DateTimeFormatter.ISO_LOCAL_DATE_TIME));
logEvent.setMessage(rs.getString("MESSAGE"));
return logEvent;
}
}
@@ -1,76 +0,0 @@
package tn.mnlr.vripper.jpa.repositories.impl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.jpa.domain.Metadata;
import tn.mnlr.vripper.jpa.repositories.IMetadataRepository;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.List;
import java.util.Optional;
@Service
public class MetadataRepository implements IMetadataRepository {
private final JdbcTemplate jdbcTemplate;
private final EventBus eventBus;
@Autowired
public MetadataRepository(JdbcTemplate jdbcTemplate, EventBus eventBus) {
this.jdbcTemplate = jdbcTemplate;
this.eventBus = eventBus;
}
@Override
public Metadata save(Metadata metadata) {
jdbcTemplate.update(
"INSERT INTO METADATA (POST_ID_REF, POST_ID, POSTED_BY, RESOLVED_NAMES) VALUES (?,?,?,?)",
metadata.getPostIdRef(),
metadata.getPostId(),
metadata.getPostedBy(),
String.join("%sep%", metadata.getResolvedNames()));
eventBus.publishEvent(Event.wrap(Event.Kind.METADATA_UPDATE, metadata.getPostIdRef()));
return metadata;
}
@Override
public Optional<Metadata> findByPostId(String postId) {
List<Metadata> metadata =
jdbcTemplate.query(
"SELECT metadata.* FROM METADATA AS metadata WHERE metadata.POST_ID = ?",
new MetadataRowMapper(),
postId);
if (metadata.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(metadata.get(0));
}
}
@Override
public int deleteByPostId(String postId) {
return jdbcTemplate.update(
"DELETE FROM METADATA AS metadata WHERE metadata.POST_ID = ?", postId);
}
}
class MetadataRowMapper implements RowMapper<Metadata> {
@Override
public Metadata mapRow(ResultSet rs, int rowNum) throws SQLException {
Metadata metadata = new Metadata();
metadata.setPostIdRef(rs.getLong("POST_ID_REF"));
metadata.setPostId(rs.getString("POST_ID"));
metadata.setPostedBy(rs.getString("POSTED_BY"));
String resolvedNames = rs.getString("RESOLVED_NAMES");
if (resolvedNames != null && !resolvedNames.isBlank()) {
metadata.setResolvedNames(List.of(resolvedNames.split("%sep%")));
}
return metadata;
}
}
@@ -1,230 +0,0 @@
package tn.mnlr.vripper.jpa.repositories.impl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.jpa.domain.Metadata;
import tn.mnlr.vripper.jpa.domain.Post;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.jpa.repositories.IPostRepository;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.util.List;
import java.util.Optional;
import java.util.Set;
@Service
public class PostRepository implements IPostRepository {
private final JdbcTemplate jdbcTemplate;
private final EventBus eventBus;
@Autowired
public PostRepository(JdbcTemplate jdbcTemplate, EventBus eventBus) {
this.jdbcTemplate = jdbcTemplate;
this.eventBus = eventBus;
}
private synchronized Long nextId() {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_POST", Long.class);
}
@Override
public Post save(Post post) {
long id = nextId();
jdbcTemplate.update(
"INSERT INTO POST (ID, DONE, FORUM, HOSTS, POST_FOLDER_NAME, POST_ID, PREVIEWS, SECURITY_TOKEN, STATUS, THANKED, THREAD_ID, THREAD_TITLE, TITLE, TOTAL, URL, ADDED_ON, RANK) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
id,
post.getDone(),
post.getForum(),
String.join(";", post.getHosts()),
post.getDownloadDirectory(),
post.getPostId(),
String.join(";", post.getPreviews()),
post.getSecurityToken(),
post.getStatus().name(),
post.isThanked(),
post.getThreadId(),
post.getThreadTitle(),
post.getTitle(),
post.getTotal(),
post.getUrl(),
Timestamp.valueOf(post.getAddedOn()),
post.getRank());
post.setId(id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return post;
}
@Override
public Optional<Post> findByPostId(String postId) {
List<Post> posts =
jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID WHERE post.POST_ID = ?",
new PostRowMapper(),
postId);
if (posts.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(posts.get(0));
}
}
@Override
public List<String> findCompleted() {
return jdbcTemplate.query(
"SELECT POST_ID FROM POST AS post WHERE status = 'COMPLETE' AND done >= total",
((rs, rowNum) -> rs.getString("POST_ID")));
}
@Override
public Optional<Post> findById(Long id) {
List<Post> posts =
jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID WHERE post.ID = ?",
new PostRowMapper(),
id);
if (posts.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(posts.get(0));
}
}
@Override
public List<Post> findAll() {
return jdbcTemplate.query(
"SELECT metadata.*,post.* FROM METADATA metadata FULL JOIN POST post ON metadata.POST_ID_REF = post.ID",
new PostRowMapper());
}
@Override
public boolean existByPostId(String postId) {
Integer count =
jdbcTemplate.queryForObject(
"SELECT COUNT(*) FROM POST AS post WHERE post.POST_ID = ?", Integer.class, postId);
if (count == null) {
return false;
} else {
return count > 0;
}
}
@Override
public int setDownloadingToStopped() {
return jdbcTemplate.update(
"UPDATE POST AS post SET post.STATUS = 'STOPPED' WHERE post.STATUS = 'DOWNLOADING' OR post.STATUS = 'PARTIAL' OR post.STATUS = 'PENDING'");
}
@Override
public int deleteByPostId(String postId) {
int mutationCount =
jdbcTemplate.update("DELETE FROM POST AS post WHERE post.POST_ID = ?", postId);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_REMOVE, postId));
return mutationCount;
}
@Override
public int updateStatus(Status status, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE POST AS post SET post.STATUS = ? WHERE post.ID = ?", status.name(), id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
@Override
public int updateDone(int done, Long id) {
int mutationCount =
jdbcTemplate.update("UPDATE POST AS post SET post.DONE = ? WHERE post.ID = ?", done, id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
@Override
public int updateFolderName(String postFolderName, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE POST AS post SET post.POST_FOLDER_NAME = ? WHERE post.ID = ?",
postFolderName,
id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
@Override
public int updateTitle(String title, Long id) {
int mutationCount =
jdbcTemplate.update("UPDATE POST AS post SET post.TITLE = ? WHERE post.ID = ?", title, id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
@Override
public int updateThanked(boolean thanked, Long id) {
int mutationCount =
jdbcTemplate.update(
"UPDATE POST AS post SET post.THANKED = ? WHERE post.ID = ?", thanked, id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
@Override
public int updateRank(int rank, Long id) {
int mutationCount =
jdbcTemplate.update("UPDATE POST AS post SET post.RANK = ? WHERE post.ID = ?", rank, id);
eventBus.publishEvent(Event.wrap(Event.Kind.POST_UPDATE, id));
return mutationCount;
}
}
class PostRowMapper implements RowMapper<Post> {
private static final String DELIMITER = ";";
@Override
public Post mapRow(ResultSet rs, int rowNum) throws SQLException {
Post post = new Post();
post.setId(rs.getLong("post.ID"));
post.setStatus(Status.valueOf(rs.getString("post.STATUS")));
post.setPostId(rs.getString("post.POST_ID"));
post.setThreadTitle(rs.getString("post.THREAD_TITLE"));
post.setThreadId(rs.getString("post.THREAD_ID"));
post.setTitle(rs.getString("post.TITLE"));
post.setUrl(rs.getString("post.URL"));
post.setDone(rs.getInt("post.DONE"));
post.setTotal(rs.getInt("post.TOTAL"));
post.setHosts(Set.of(rs.getString("post.HOSTS").split(DELIMITER)));
post.setForum(rs.getString("post.FORUM"));
post.setSecurityToken(rs.getString("post.SECURITY_TOKEN"));
post.setDownloadDirectory(rs.getString("post.POST_FOLDER_NAME"));
post.setThanked(rs.getBoolean("post.THANKED"));
String previews;
if ((previews = rs.getString("post.PREVIEWS")) != null) {
post.setPreviews(Set.of(previews.split(DELIMITER)));
}
post.setAddedOn(rs.getTimestamp("post.ADDED_ON").toLocalDateTime());
post.setRank(rs.getInt("post.RANK"));
Long metadataId = rs.getLong("metadata.POST_ID_REF");
if (!rs.wasNull()) {
Metadata metadata = new Metadata();
metadata.setPostIdRef(metadataId);
metadata.setPostId(rs.getString("metadata.POST_ID"));
String resolvedNames = rs.getString("metadata.RESOLVED_NAMES");
if (resolvedNames != null && !resolvedNames.isBlank()) {
metadata.setResolvedNames(List.of(resolvedNames.split("%sep%")));
}
metadata.setPostedBy(rs.getString("metadata.POSTED_BY"));
post.setMetadata(metadata);
}
return post;
}
}
@@ -1,107 +0,0 @@
package tn.mnlr.vripper.jpa.repositories.impl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.jpa.domain.Queued;
import tn.mnlr.vripper.jpa.repositories.IQueuedRepository;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.List;
import java.util.Optional;
@Service
public class QueuedRepository implements IQueuedRepository {
private final JdbcTemplate jdbcTemplate;
private final EventBus eventBus;
@Autowired
public QueuedRepository(JdbcTemplate jdbcTemplate, EventBus eventBus) {
this.jdbcTemplate = jdbcTemplate;
this.eventBus = eventBus;
}
private synchronized Long nextId() {
return jdbcTemplate.queryForObject("CALL NEXT VALUE FOR SEQ_QUEUED", Long.class);
}
@Override
public Queued save(Queued queued) {
long id = nextId();
jdbcTemplate.update(
"INSERT INTO QUEUED (ID, TOTAL, LINK, LOADING, POST_ID, THREAD_ID) values (?,?,?,?,?,?)",
id,
queued.getTotal(),
queued.getLink(),
queued.isLoading(),
queued.getPostId(),
queued.getThreadId());
queued.setId(id);
eventBus.publishEvent(Event.wrap(Event.Kind.QUEUED_UPDATE, id));
return queued;
}
@Override
public Optional<Queued> findByThreadId(String threadId) {
List<Queued> queuedList =
jdbcTemplate.query(
"SELECT * FROM QUEUED AS queued WHERE queued.THREAD_ID = ?",
new QueuedRowMapper(),
threadId);
if (queuedList.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(queuedList.get(0));
}
}
@Override
public List<Queued> findAll() {
return jdbcTemplate.query("SELECT * FROM QUEUED", new QueuedRowMapper());
}
@Override
public Optional<Queued> findById(Long id) {
List<Queued> queuedList =
jdbcTemplate.query(
"SELECT * FROM QUEUED AS queued WHERE queued.ID = ?", new QueuedRowMapper(), id);
if (queuedList.isEmpty()) {
return Optional.empty();
} else {
return Optional.of(queuedList.get(0));
}
}
@Override
public int deleteByThreadId(String threadId) {
int mutationCount =
jdbcTemplate.update("DELETE FROM QUEUED AS queued WHERE THREAD_ID = ?", threadId);
eventBus.publishEvent(Event.wrap(Event.Kind.QUEUED_REMOVE, threadId));
return mutationCount;
}
@Override
public void deleteAll() {
jdbcTemplate.update("DELETE FROM QUEUED");
}
}
class QueuedRowMapper implements RowMapper<Queued> {
@Override
public Queued mapRow(ResultSet rs, int rowNum) throws SQLException {
Queued queued = new Queued();
queued.setId(rs.getLong("ID"));
queued.setLink(rs.getString("LINK"));
queued.setThreadId(rs.getString("THREAD_ID"));
queued.setPostId(rs.getString("POST_ID"));
queued.setTotal(rs.getInt("TOTAL"));
queued.setLoading(rs.getBoolean("LOADING"));
return queued;
}
}
@@ -0,0 +1,48 @@
package tn.mnlr.vripper.q;
import lombok.Getter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import tn.mnlr.vripper.entities.Image;
import java.util.Objects;
import java.util.concurrent.Callable;
public class DownloadJob implements Callable<Image> {
private static final Logger logger = LoggerFactory.getLogger(DownloadJob.class);
@Getter
private final Image image;
@Getter
private final ImageFileData imageFileData = new ImageFileData();
public DownloadJob(Image image) {
this.image = image;
}
@Override
public Image call() throws Exception {
logger.info(String.format("Starting downloading %s", image.getUrl()));
image.setStatus(Image.Status.DOWNLOADING);
image.setCurrent(0);
image.getHost().download(image, imageFileData);
image.setStatus(Image.Status.COMPLETE);
return image;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
DownloadJob that = (DownloadJob) o;
return image.equals(that.image);
}
@Override
public int hashCode() {
return Objects.hash(image);
}
}
@@ -0,0 +1,143 @@
package tn.mnlr.vripper.q;
import lombok.Getter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.entities.Post;
import tn.mnlr.vripper.services.AppStateService;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.stream.Collectors;
@Service
public class DownloadQ {
private static final List<Post.Status> FINISHED = Arrays.asList(Post.Status.ERROR, Post.Status.COMPLETE, Post.Status.STOPPED);
private static final Logger logger = LoggerFactory.getLogger(DownloadQ.class);
@Autowired
private AppStateService appStateService;
@Autowired
private ExecutionService executionService;
private BlockingQueue<DownloadJob> downloadQ = new LinkedBlockingQueue<>();
@Getter
private boolean notPauseQ = true;
public synchronized void put(Image image) throws InterruptedException {
logger.info(String.format("Enqueuing a job for %s", image.getUrl()));
DownloadJob downloadJob = new DownloadJob(image);
downloadQ.put(downloadJob);
appStateService.newDownloadJob(downloadJob);
}
public DownloadJob take() throws InterruptedException {
DownloadJob downloadJob = downloadQ.take();
logger.info(String.format("Retrieving a job for %s", downloadJob.getImage().getUrl()));
return downloadJob;
}
public void enqueue(Post post) throws InterruptedException {
for (Image image : post.getImages()) {
put(image);
}
}
public void restart(String postId) throws InterruptedException {
if (appStateService.getRunningPosts().get(postId) != null && appStateService.getRunningPosts().get(postId).get() > 0) {
logger.warn(String.format("Cannot restart, jobs are currently running for post id %s", postId));
return;
}
List<Image> images = appStateService.getPost(postId)
.getImages()
.stream()
.filter(e -> !e.getStatus().equals(Image.Status.COMPLETE))
.collect(Collectors.toList());
if (images.isEmpty()) {
return;
}
appStateService.getPost(postId).setStatus(Post.Status.PENDING);
logger.info(String.format("Restarting %d jobs for post id %s", images.size(), postId));
for (Image image : images) {
image.init();
put(image);
}
}
public void removeScheduled(Image image) {
image.setStatus(Image.Status.STOPPED);
logger.info(String.format("Removing scheduled job for %s", image.getUrl()));
Iterator<DownloadJob> iterator = downloadQ.iterator();
boolean removed = false;
while(iterator.hasNext()) {
DownloadJob next = iterator.next();
if(next.getImage().getPostId().equals(image.getPostId())) {
iterator.remove();
appStateService.doneDownloadJob(image);
logger.info(String.format("Scheduled job for %s is removed", image.getUrl()));
removed = true;
break;
}
}
if(!removed) {
logger.warn(String.format("Job for %s does not exist", image.getUrl()));
}
}
public void removeRunning(String postId) {
logger.info(String.format("Interrupting running jobs for post id %s", postId));
executionService.stop(postId);
}
public synchronized void stop(String postId) {
try {
if (FINISHED.contains(appStateService.getPost(postId).getStatus())) {
return;
}
notPauseQ = false;
appStateService.getPost(postId).setStatus(Post.Status.STOPPED);
List<Image> images = appStateService.getPost(postId)
.getImages()
.stream()
.filter(e -> !e.getStatus().equals(Image.Status.COMPLETE))
.collect(Collectors.toList());
if (images.isEmpty()) {
return;
}
logger.info(String.format("Stopping %d jobs for post id %s", images.size(), postId));
images.forEach(image -> removeScheduled(image));
removeRunning(postId);
} finally {
notPauseQ = true;
}
}
public int size() {
return downloadQ.size();
}
public synchronized void stopAll() {
appStateService.getCurrentPosts().values().stream().map(Post::getPostId).forEach(this::stop);
}
public synchronized void restartAll() throws InterruptedException {
for (Post post : appStateService.getCurrentPosts().values()) {
String postId = post.getPostId();
restart(postId);
}
}
}
@@ -0,0 +1,159 @@
package tn.mnlr.vripper.q;
import net.jodah.failsafe.Failsafe;
import net.jodah.failsafe.FailsafeException;
import net.jodah.failsafe.RetryPolicy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.services.AppSettingsService;
import tn.mnlr.vripper.services.AppStateService;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
@Service
public class ExecutionService {
private static final Logger logger = LoggerFactory.getLogger(ExecutionService.class);
@Autowired
private DownloadQ downloadQ;
@Autowired
private AppSettingsService settings;
@Autowired
private AppStateService appStateService;
private AtomicInteger threadCount = new AtomicInteger();
private ExecutorService executor = Executors.newFixedThreadPool(10);
private Thread executionThread;
private RetryPolicy<Object> retryPolicy;
private List<DownloadJob> running = Collections.synchronizedList(new ArrayList<>());
private Map<String, Future<?>> futures = new ConcurrentHashMap<>();
@PostConstruct
private void init() {
retryPolicy = new RetryPolicy<>()
.handleIf(e -> !(e instanceof InterruptedException))
.withDelay(Duration.ofSeconds(20))
.withMaxRetries(2)
.abortOn(InterruptedException.class)
.onFailedAttempt(e -> logger.warn(String.format("#%d tries failed", e.getAttemptCount()), e.getLastFailure()));
executionThread = new Thread(this::start, "Executor thread");
executionThread.start();
}
@PreDestroy
private void destroy() throws InterruptedException {
logger.info("Shutting down ExecutionService");
executionThread.interrupt();
executor.shutdown();
appStateService.getCurrentPosts().keySet().forEach(p -> {
logger.info(String.format("Stopping download jobs for %s", p));
this.stop(p);
});
executor.awaitTermination(10, TimeUnit.SECONDS);
}
public void stop(String postId) {
List<DownloadJob> data = running
.stream()
.filter(e -> e.getImage().getPostId().equals(postId))
.peek(e -> e.getImage().setStatus(Image.Status.STOPPED))
.collect(Collectors.toList());
logger.warn(String.format("Interrupting %d jobs for post id %s", data.size(), postId));
data.forEach(e -> {
futures.get(e.getImage().getUrl()).cancel(true);
if(e.getImageFileData().getImageRequest() != null) {
e.getImageFileData().getImageRequest().abort();
}
});
}
boolean canRun() {
boolean canRun = threadCount.get() < settings.getMaxThreads();
if (canRun && downloadQ.isNotPauseQ()) {
threadCount.incrementAndGet();
return true;
}
return false;
}
public void start() {
while (!Thread.interrupted()) {
if (canRun()) {
DownloadJob take = null;
try {
take = downloadQ.take();
if (take == null) {
continue;
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
DownloadJob finalTake = take;
Runnable task = () -> {
running.add(finalTake);
Failsafe.with(retryPolicy)
.onFailure(e -> {
if (e.getFailure() instanceof InterruptedException || (e.getFailure() instanceof FailsafeException && e.getFailure().getCause() instanceof InterruptedException)) {
logger.info("Job successfully interrupted");
return;
}
logger.error(String.format("Failed to download %s after %d tries", finalTake.getImage().getUrl(), e.getAttemptCount()), e.getFailure());
finalTake.getImage().setStatus(Image.Status.ERROR);
})
.onComplete(e -> {
appStateService.doneDownloadJob(finalTake.getImage());
logger.info(String.format("Finished downloading %s", finalTake.getImage().getUrl()));
synchronized (threadCount) {
threadCount.decrementAndGet();
running.remove(finalTake);
futures.remove(finalTake.getImage().getUrl());
threadCount.notify();
}
})
.get(finalTake::call);
};
logger.info(String.format("Scheduling a job for %s", finalTake.getImage().getUrl()));
futures.put(finalTake.getImage().getUrl(), executor.submit(task));
} else {
synchronized (threadCount) {
try {
threadCount.wait(2_000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
}
}
}
public int runningCount() {
return running.size();
}
}
@@ -0,0 +1,15 @@
package tn.mnlr.vripper.q;
import lombok.Getter;
import lombok.Setter;
import org.apache.http.client.methods.HttpUriRequest;
@Getter
@Setter
public class ImageFileData {
private String pageUrl;
private String imageName;
private String imageUrl;
private HttpUriRequest imageRequest;
}
@@ -0,0 +1,136 @@
package tn.mnlr.vripper.services;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Getter;
import lombok.Setter;
import org.apache.commons.codec.digest.DigestUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.exception.ValidationException;
import javax.annotation.PreDestroy;
import java.nio.file.Files;
import java.nio.file.InvalidPathException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.prefs.BackingStoreException;
import java.util.prefs.Preferences;
@Service
@Getter
@Setter
public class AppSettingsService {
private static final Logger logger = LoggerFactory.getLogger(AppSettingsService.class);
private Preferences prefs = Preferences.userNodeForPackage(AppSettingsService.class);
private final String DOWNLOAD_PATH = "DOWNLOAD_PATH";
private final String MAX_THREADS = "MAX_THREADS";
private final String AUTO_START = "AUTO_START";
private final String V_LOGIN = "VLOGIN";
private final String V_USERNAME = "VUSERNAME";
private final String V_PASSWORD = "VPASSWORD";
private final String V_THANKS = "VTHANKS";
private final String DESKTOP_CLIPBOARD = "DESKTOP_CLIPBOARD";
private String downloadPath;
private int maxThreads;
private boolean autoStart;
private boolean vLogin;
private String vUsername;
private String vPassword;
private boolean vThanks;
private boolean desktopClipboard;
public void setVPassword(String vPassword) {
if(vPassword.isEmpty()) {
this.vPassword = "";
} else {
this.vPassword = DigestUtils.md5Hex(vPassword);
}
}
public void restore() {
downloadPath = prefs.get(DOWNLOAD_PATH, System.getProperty("user.dir"));
maxThreads = prefs.getInt(MAX_THREADS, 4);
autoStart = prefs.getBoolean(AUTO_START, true);
vLogin = prefs.getBoolean(V_LOGIN, false);
vUsername = prefs.get(V_USERNAME, "");
vPassword = prefs.get(V_PASSWORD, "");
vThanks = prefs.getBoolean(V_THANKS, false);
desktopClipboard = prefs.getBoolean(DESKTOP_CLIPBOARD, false);
}
@PreDestroy
public void save() {
prefs.put(DOWNLOAD_PATH, downloadPath);
prefs.putInt(MAX_THREADS, maxThreads);
prefs.putBoolean(AUTO_START, autoStart);
prefs.putBoolean(V_LOGIN, vLogin);
prefs.put(V_USERNAME, vUsername);
prefs.put(V_PASSWORD, vPassword);
prefs.putBoolean(V_THANKS, vThanks);
prefs.putBoolean(DESKTOP_CLIPBOARD, desktopClipboard);
try {
prefs.sync();
} catch (BackingStoreException e) {
logger.error("Failed to store user settings", e);
}
}
public void check(Settings settings) throws ValidationException {
Path path;
try {
path = Paths.get(settings.getDownloadPath());
} catch (InvalidPathException e) {
throw new ValidationException(String.format("%s is invalid", settings.getDownloadPath()));
}
if (!Files.exists(path)) {
throw new ValidationException(String.format("%s does not exist", settings.getDownloadPath()));
} else if (!Files.isDirectory(path)) {
throw new ValidationException(String.format("%s is not a directory", settings.getDownloadPath()));
}
if (settings.getMaxThreads() < 1 || settings.getMaxThreads() > 8) {
throw new ValidationException(String.format("Invalid max concurrent download settings, values must be in [%d,%d]", 1, 8));
}
}
@Getter
public static class Settings {
@JsonProperty("downloadPath")
private String downloadPath;
@JsonProperty("maxThreads")
private int maxThreads;
@JsonProperty("autoStart")
private boolean autoStart;
@JsonProperty("vLogin")
private boolean vLogin;
@JsonProperty("vUsername")
private String vUsername;
@JsonProperty("vPassword")
private String vPassword;
@JsonProperty("vThanks")
private boolean vThanks;
@JsonProperty("desktopClipboard")
private boolean desktopClipboard;
public Settings(String downloadPath, int maxThreads, boolean autoStart, boolean vLogin, String vUsername, String vPassword, boolean vThanks, boolean desktopClipboard) {
this.downloadPath = downloadPath;
this.maxThreads = maxThreads;
this.autoStart = autoStart;
this.vLogin = vLogin;
this.vUsername = vUsername;
this.vPassword = vPassword;
this.vThanks = vThanks;
this.desktopClipboard = desktopClipboard;
}
}
}
@@ -0,0 +1,130 @@
package tn.mnlr.vripper.services;
import io.reactivex.processors.PublishProcessor;
import lombok.Getter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.entities.Post;
import tn.mnlr.vripper.host.Host;
import tn.mnlr.vripper.q.DownloadJob;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
@Service
@Getter
public class AppStateService {
private Map<String, Image> currentImages = new ConcurrentHashMap<>();
private Map<String, Post> currentPosts = new ConcurrentHashMap<>();
private Map<String, AtomicInteger> runningPosts = new ConcurrentHashMap<>();
private PublishProcessor<Image> liveImageUpdates = PublishProcessor.create();
private PublishProcessor<Post> livePostsState = PublishProcessor.create();
@Autowired
private PersistenceService persistenceService;
public void onImageUpdate(Image imageState) {
persistenceService.getProcessor().onNext(currentPosts);
liveImageUpdates.onNext(imageState);
if (imageState.isCompleted()) {
Post postState = currentPosts.get(imageState.getPostId());
postState.increase();
}
}
public Image createImage(String pageUrl, String postId, String postName, Host host) {
return new Image(pageUrl, postId, postName, host, this);
}
public Post createPost(String title, String url, List<Image> images, Map<String, String> metadata, String postId, String postCounter) {
return new Post(title, url, images, metadata, postId, postCounter, this);
}
public Post getPost(String postId) {
return currentPosts.get(postId);
}
public synchronized void newDownloadJob(DownloadJob downloadJob) {
String postId = downloadJob.getImage().getPostId();
checkKeyRunningPosts(postId);
int i = runningPosts.get(postId).incrementAndGet();
if (i > 0) {
Post post = currentPosts.get(postId);
post.setStatus(Post.Status.DOWNLOADING);
livePostsState.onNext(post);
}
}
public void doneDownloadJob(Image image) {
String postId = image.getPostId();
int i = runningPosts.get(postId).decrementAndGet();
Post post = currentPosts.get(postId);
if(image.getStatus().equals(Image.Status.ERROR)) {
post.setStatus(Post.Status.PARTIAL);
}
if (i == 0) {
if (post.getImages().stream().map(Image::getStatus).filter(e -> e.equals(Image.Status.ERROR)).count() > 0) {
post.setStatus(Post.Status.ERROR);
} else {
if (!Post.Status.STOPPED.equals(post.getStatus())) {
post.setStatus(Post.Status.COMPLETE);
}
}
livePostsState.onNext(post);
}
}
private synchronized void checkKeyRunningPosts(String key) {
if (!runningPosts.containsKey(key)) {
runningPosts.put(key, new AtomicInteger(0));
}
}
public synchronized void remove(String postId) {
currentPosts.get(postId).setRemoved(true);
runningPosts.remove(postId);
currentPosts.remove(postId);
Iterator<Map.Entry<String, Image>> iterator = currentImages.entrySet().iterator();
while (iterator.hasNext()) {
Map.Entry<String, Image> entry = iterator.next();
if(entry.getValue().getPostId().equals(postId)) {
iterator.remove();
}
}
persistenceService.getProcessor().onNext(currentPosts);
}
public synchronized List<String> clearAll() {
List<String> toRemove = this.currentPosts
.values()
.stream()
.filter(e -> e.getStatus().equals(Post.Status.COMPLETE) && e.getDone().get() >= e.getTotal())
.map(Post::getPostId)
.collect(Collectors.toList());
toRemove.forEach(this::remove);
return toRemove;
}
public synchronized List<String> removeAll() {
List<String> toRemove = this.currentPosts
.values()
.stream()
.map(Post::getPostId)
.collect(Collectors.toList());
toRemove.forEach(this::remove);
return toRemove;
}
}
@@ -0,0 +1,54 @@
package tn.mnlr.vripper.services;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.stereotype.Service;
@Service
public class ConnectionManager {
private ConnectionManager() {
buildConnectionPool();
}
private PoolingHttpClientConnectionManager pcm;
private RequestConfig rc = RequestConfig.custom()
.setConnectionRequestTimeout(10_000)
.setConnectTimeout(10_000)
.setSocketTimeout(10_000)
.setCookieSpec(CookieSpecs.STANDARD)
.build();
private void buildConnectionPool() {
pcm = new PoolingHttpClientConnectionManager();
pcm.setMaxTotal(200);
pcm.setDefaultMaxPerRoute(10);
}
public HttpClientBuilder getClient() {
return HttpClients.custom()
.setConnectionManager(pcm)
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.setDefaultRequestConfig(rc);
}
public HttpGet buildHttpGet(String url) {
HttpGet httpGet = new HttpGet(url);
httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
return httpGet;
}
public HttpPost buildHttpPost(String url) {
HttpPost httpPost = new HttpPost(url);
httpPost.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
return httpPost;
}
}
@@ -1,154 +0,0 @@
package tn.mnlr.vripper.services;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import net.jodah.failsafe.RetryPolicy;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.AbstractExecutionAwareRequest;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.client.LaxRedirectStrategy;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import reactor.core.Disposable;
import tn.mnlr.vripper.download.DownloadJob;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.services.domain.Settings;
import javax.annotation.PreDestroy;
import java.net.URI;
import java.time.temporal.ChronoUnit;
import java.util.List;
import java.util.concurrent.TimeUnit;
@Service
@EnableScheduling
@Slf4j
public class ConnectionService {
private final Disposable disposable;
private PoolingHttpClientConnectionManager pcm;
private RequestConfig rc;
@Getter private RetryPolicy<Object> retryPolicy;
private int connectionTimeout;
private int maxAttempts;
public ConnectionService(EventBus eventBus, SettingsService settingsService) {
connectionTimeout = settingsService.getSettings().getConnectionTimeout();
maxAttempts = settingsService.getSettings().getMaxAttempts();
disposable =
eventBus
.flux()
.filter(e -> e.getKind().equals(Event.Kind.SETTINGS_UPDATE))
.map(e -> ((Settings) e.getData()))
.subscribe(
settings -> {
if (connectionTimeout != settings.getConnectionTimeout()) {
connectionTimeout = settings.getConnectionTimeout();
buildRequestConfig();
}
if (maxAttempts != settings.getMaxAttempts()) {
maxAttempts = settings.getMaxAttempts();
buildRetryPolicy();
}
});
buildRequestConfig();
buildRetryPolicy();
buildConnectionPool();
}
@PreDestroy
private void destroy() {
disposable.dispose();
}
private void buildRetryPolicy() {
retryPolicy =
new RetryPolicy<>()
.withDelay(2, 5, ChronoUnit.SECONDS)
.withMaxAttempts(maxAttempts)
.onFailedAttempt(
e ->
log.warn(
String.format("#%d tries failed", e.getAttemptCount()),
e.getLastFailure()));
}
private void buildConnectionPool() {
pcm = new PoolingHttpClientConnectionManager();
pcm.setMaxTotal(Integer.MAX_VALUE);
pcm.setDefaultMaxPerRoute(Integer.MAX_VALUE);
}
private void buildRequestConfig() {
rc =
RequestConfig.custom()
.setConnectionRequestTimeout(connectionTimeout * 1000)
.setConnectTimeout(connectionTimeout * 1000)
.setSocketTimeout(connectionTimeout * 1000)
.setCookieSpec(CookieSpecs.STANDARD)
.build();
}
@Scheduled(fixedDelay = 5_000)
private void idleConnectionMonitoring() {
pcm.closeExpiredConnections();
pcm.closeIdleConnections(30, TimeUnit.SECONDS);
}
public HttpClientBuilder getClient() {
return HttpClients.custom()
.setConnectionManager(pcm)
.setRedirectStrategy(new LaxRedirectStrategy())
.disableAutomaticRetries()
.setDefaultRequestConfig(rc);
}
public HttpGet buildHttpGet(String url, final HttpClientContext context) {
HttpGet httpGet = new HttpGet(url.replace(" ", "+"));
httpGet.addHeader(
"User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
addToContext(context, httpGet);
return httpGet;
}
public HttpPost buildHttpPost(String url, final HttpClientContext context) {
HttpPost httpPost = new HttpPost(url.replace(" ", "+"));
httpPost.addHeader(
"User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
addToContext(context, httpPost);
return httpPost;
}
public HttpGet buildHttpGet(URI uri, final HttpClientContext context) {
HttpGet httpGet = new HttpGet(uri);
httpGet.addHeader(
"User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
addToContext(context, httpGet);
return httpGet;
}
public void addToContext(HttpClientContext context, AbstractExecutionAwareRequest request) {
if (context != null) {
List<AbstractExecutionAwareRequest> requests =
(List<AbstractExecutionAwareRequest>)
context.getAttribute(DownloadJob.ContextAttributes.OPEN_CONNECTION.toString());
if (requests != null) {
requests.add(request);
}
}
}
}
@@ -1,244 +0,0 @@
package tn.mnlr.vripper.services;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import tn.mnlr.vripper.jpa.domain.*;
import tn.mnlr.vripper.jpa.domain.enums.Status;
import tn.mnlr.vripper.jpa.repositories.IImageRepository;
import tn.mnlr.vripper.jpa.repositories.IMetadataRepository;
import tn.mnlr.vripper.jpa.repositories.IPostRepository;
import tn.mnlr.vripper.jpa.repositories.IQueuedRepository;
import tn.mnlr.vripper.jpa.repositories.impl.LogEventRepository;
import javax.annotation.PostConstruct;
import java.util.*;
import java.util.stream.Collectors;
@Service
@Transactional
@Slf4j
public class DataService {
private final IPostRepository postRepository;
private final IImageRepository imageRepository;
private final IQueuedRepository queuedRepository;
private final IMetadataRepository metadataRepository;
private final SettingsService settingsService;
private final LogEventRepository eventRepository;
@Autowired
public DataService(
IPostRepository postRepository,
IImageRepository imageRepository,
IQueuedRepository queuedRepository,
IMetadataRepository metadataRepository,
SettingsService settingsService,
LogEventRepository eventRepository) {
this.postRepository = postRepository;
this.imageRepository = imageRepository;
this.queuedRepository = queuedRepository;
this.metadataRepository = metadataRepository;
this.settingsService = settingsService;
this.eventRepository = eventRepository;
}
private void save(Post post) {
postRepository.save(post);
}
private void save(Queued queued) {
queuedRepository.save(queued);
}
private void save(Image image) {
imageRepository.save(image);
}
public boolean exists(String postId) {
return postRepository.existByPostId(postId);
}
public void newPost(Post post, Collection<Image> images) {
save(post);
images.forEach(
image -> {
image.setPostIdRef(post.getId());
save(image);
});
sortPostsByRank();
}
public synchronized void afterJobFinish(Image image, Post post) {
if (image.getStatus().equals(Status.COMPLETE)) {
post.setDone(post.getDone() + 1);
updatePostDone(post.getDone(), post.getId());
} else if (image.getStatus().equals(Status.ERROR)) {
post.setStatus(Status.PARTIAL);
updatePostStatus(post.getStatus(), post.getId());
}
}
private void updatePostDone(int done, Long id) {
postRepository.updateDone(done, id);
}
public void updatePostStatus(Status status, Long id) {
postRepository.updateStatus(status, id);
}
public void updatePostThanks(boolean thanked, Long id) {
postRepository.updateThanked(thanked, id);
}
public void updatePostTitle(String title, Long id) {
postRepository.updateTitle(title, id);
}
public void updatePostDownloadDirectory(String postFolderName, Long id) {
postRepository.updateFolderName(postFolderName, id);
}
public void updatePostRank(int rank, Long id) {
postRepository.updateRank(rank, id);
}
public void finishPost(@NonNull Post post) {
if (!imageRepository.findByPostIdAndIsError(post.getPostId()).isEmpty()) {
post.setStatus(Status.ERROR);
updatePostStatus(post.getStatus(), post.getId());
} else {
if (post.getDone() < post.getTotal()) {
post.setStatus(Status.STOPPED);
updatePostStatus(post.getStatus(), post.getId());
} else {
post.setStatus(Status.COMPLETE);
updatePostStatus(post.getStatus(), post.getId());
if (settingsService.getSettings().getClearCompleted()) {
remove(List.of(post.getPostId()));
}
}
}
}
private void remove(@NonNull final List<String> postIds) {
for (String postId : postIds) {
imageRepository.deleteAllByPostId(postId);
metadataRepository.deleteByPostId(postId);
postRepository.deleteByPostId(postId);
}
sortPostsByRank();
}
public void newQueueLink(@NonNull final Queued queued) {
save(queued);
}
public void removeQueueLink(@NonNull final String threadId) {
queuedRepository.deleteByThreadId(threadId);
}
public List<String> clearCompleted() {
List<String> completed = postRepository.findCompleted();
remove(completed);
return completed;
}
public void removeAll(final List<String> postIds) {
remove(
Objects.requireNonNullElse(
postIds,
postRepository.findAll().stream().map(Post::getPostId).collect(Collectors.toList())));
}
public List<Image> findByPostIdAndIsNotCompleted(@NonNull String postId) {
return imageRepository.findByPostIdAndIsNotCompleted(postId);
}
public long countErrorImages() {
return imageRepository.countError();
}
public List<Image> findImagesByPostId(String postId) {
return imageRepository.findByPostId(postId);
}
public List<Post> findAllPosts() {
return postRepository.findAll();
}
public Optional<Post> findPostByPostId(String postId) {
return postRepository.findByPostId(postId);
}
public void stopImagesByPostIdAndIsNotCompleted(String postId) {
imageRepository.stopByPostIdAndIsNotCompleted(postId);
}
public Optional<Queued> findQueuedByThreadId(String threadId) {
return queuedRepository.findByThreadId(threadId);
}
public List<Queued> findAllQueued() {
return queuedRepository.findAll();
}
public Optional<Post> findById(Long aLong) {
return postRepository.findById(aLong);
}
public Optional<Image> findImageById(Long aLong) {
return imageRepository.findById(aLong);
}
public Optional<Queued> findQueuedById(Long aLong) {
return queuedRepository.findById(aLong);
}
public synchronized void setMetadata(Post post, Metadata metadata) {
if (metadataRepository.findByPostId(post.getPostId()).isEmpty()) {
metadata.setPostIdRef(post.getId());
metadataRepository.save(metadata);
}
}
public Optional<Metadata> findMetadataByPostId(String postId) {
return metadataRepository.findByPostId(postId);
}
public void updateImageStatus(Status status, Long id) {
imageRepository.updateStatus(status, id);
}
public void updateImageCurrent(long current, Long id) {
imageRepository.updateCurrent(current, id);
}
public void updateImageTotal(long total, Long id) {
imageRepository.updateTotal(total, id);
}
public Optional<LogEvent> findEventById(Long id) {
return eventRepository.findById(id);
}
public List<LogEvent> findAllEvents() {
return eventRepository.findAll();
}
public void clearQueueLinks() {
queuedRepository.deleteAll();
}
public synchronized void sortPostsByRank() {
List<Post> posts = findAllPosts();
posts.sort(Comparator.comparing(Post::getAddedOn));
for (int i = 0; i < posts.size(); i++) {
posts.get(i).setRank(i);
updatePostRank(i, posts.get(i).getId());
}
}
}
@@ -0,0 +1,26 @@
package tn.mnlr.vripper.services;
import lombok.Getter;
@Getter
public class DownloadSpeed {
private final String type = "downSpeed";
private String speed;
public DownloadSpeed(long bytes) {
speed = formatSI(bytes);
}
private String formatSI(long bytes) {
return humanReadableByteCount(bytes, true);
}
private String humanReadableByteCount(long bytes, boolean si) {
int unit = si ? 1000 : 1024;
if (bytes < unit) return bytes + " B";
int exp = (int) (Math.log(bytes) / Math.log(unit));
String pre = (si ? "kMGTPE" : "KMGTPE").charAt(exp - 1) + (si ? "" : "i");
return String.format("%.1f %sB", bytes / Math.pow(unit, exp), pre);
}
}
@@ -1,11 +1,10 @@
package tn.mnlr.vripper.services;
import io.reactivex.processors.PublishProcessor;
import lombok.Getter;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import java.util.concurrent.atomic.AtomicLong;
@@ -13,29 +12,17 @@ import java.util.concurrent.atomic.AtomicLong;
@EnableScheduling
public class DownloadSpeedService {
private final AtomicLong read = new AtomicLong(0);
private final EventBus eventBus;
@Getter private long currentValue;
private boolean allowWrite = false;
private AtomicLong read = new AtomicLong(0);
public DownloadSpeedService(EventBus eventBus) {
this.eventBus = eventBus;
}
@Getter
private PublishProcessor<Long> readBytesPerSecond = PublishProcessor.create();
public void increase(long read) {
if (allowWrite) {
this.read.addAndGet(read);
public void increase(long read) {
this.read.addAndGet(read);
}
}
@Scheduled(fixedDelay = 1000)
private void calc() {
allowWrite = false;
long newValue = read.getAndSet(0);
if (newValue != currentValue) {
currentValue = newValue;
eventBus.publishEvent(Event.wrap(Event.Kind.BYTES_PER_SECOND, currentValue));
@Scheduled(fixedDelay = 1000)
private void calc() {
readBytesPerSecond.onNext(read.getAndSet(0));
}
allowWrite = true;
}
}
@@ -0,0 +1,39 @@
package tn.mnlr.vripper.services;
import lombok.Getter;
import java.util.Objects;
@Getter
public class GlobalState {
private final String type = "globalState";
private long running;
private long queued;
private long remaining;
private long error;
public GlobalState(long running, long queued, long remaining, long error) {
this.running = running;
this.queued = queued;
this.remaining = remaining;
this.error = error;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
GlobalState that = (GlobalState) o;
return running == that.running &&
queued == that.queued &&
remaining == that.remaining &&
error == that.error &&
Objects.equals(type, that.type);
}
@Override
public int hashCode() {
return Objects.hash(type, running, queued, remaining, error);
}
}
@@ -1,42 +1,52 @@
package tn.mnlr.vripper.services;
import io.reactivex.processors.PublishProcessor;
import lombok.Getter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import tn.mnlr.vripper.download.DownloadService;
import tn.mnlr.vripper.event.Event;
import tn.mnlr.vripper.event.EventBus;
import tn.mnlr.vripper.services.domain.GlobalState;
import tn.mnlr.vripper.entities.Image;
import tn.mnlr.vripper.q.DownloadQ;
import tn.mnlr.vripper.q.ExecutionService;
@Service
@EnableScheduling
public class GlobalStateService {
private final DownloadService downloadService;
private final DataService dataService;
private final EventBus eventBus;
@Getter private GlobalState currentState;
@Autowired
private DownloadQ downloadQ;
@Autowired
public GlobalStateService(
DownloadService downloadService, DataService dataService, EventBus eventBus) {
this.downloadService = downloadService;
this.dataService = dataService;
this.eventBus = eventBus;
}
@Autowired
private ExecutionService executionService;
@Scheduled(fixedDelay = 3000)
private void interval() {
GlobalState newGlobalState =
new GlobalState(
downloadService.runningCount(),
downloadService.pendingCount(),
dataService.countErrorImages());
if (!newGlobalState.equals(currentState)) {
currentState = newGlobalState;
eventBus.publishEvent(Event.wrap(Event.Kind.GLOBAL_STATE, currentState));
@Autowired
private AppStateService appStateService;
@Getter
private GlobalState currentState;
@Getter
private PublishProcessor<GlobalState> liveGlobalState = PublishProcessor.create();
@Scheduled(fixedDelay = 3000)
private void interval() {
GlobalState newGlobalState = new GlobalState(
executionService.runningCount(),
downloadQ.size(),
appStateService.getCurrentImages()
.values()
.stream()
.filter(e -> e.getTotal() == 0 || e.getTotal() != e.getCurrent().get())
.count(),
appStateService.getCurrentImages()
.values()
.stream()
.filter(e -> e.getStatus().equals(Image.Status.ERROR))
.count());
if (!newGlobalState.equals(currentState)) {
currentState = newGlobalState;
liveGlobalState.onNext(currentState);
}
}
}
}
@@ -1,113 +0,0 @@
package tn.mnlr.vripper.services;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.util.EntityUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;
import tn.mnlr.vripper.exception.HostException;
import tn.mnlr.vripper.exception.HtmlProcessorException;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
@Service
@Slf4j
public class HostService {
private final ConnectionService cm;
private final HtmlProcessorService htmlProcessorService;
@Autowired
public HostService(ConnectionService cm, HtmlProcessorService htmlProcessorService) {
this.cm = cm;
this.htmlProcessorService = htmlProcessorService;
}
public Response getResponse(final String url, final HttpClientContext context)
throws HostException {
String basePage;
HttpClient client = cm.getClient().build();
HttpGet httpGet = cm.buildHttpGet(url, context);
Header[] headers;
log.debug(String.format("Requesting %s", url));
try (CloseableHttpResponse response =
(CloseableHttpResponse) client.execute(httpGet, context)) {
if (response.getStatusLine().getStatusCode() / 100 != 2) {
throw new HostException(
String.format(
"Unexpected response code: %d", response.getStatusLine().getStatusCode()));
}
headers = response.getAllHeaders();
basePage = EntityUtils.toString(response.getEntity());
log.debug(String.format("%s response: %n%s", url, basePage));
EntityUtils.consumeQuietly(response.getEntity());
} catch (IOException e) {
throw new HostException(e);
}
try {
log.debug(String.format("Cleaning %s response", url));
return new Response(htmlProcessorService.clean(basePage), headers);
} catch (HtmlProcessorException e) {
throw new HostException(e);
}
}
public String appendUri(String uri, String appendQuery) throws URISyntaxException {
URI oldUri = new URI(uri);
String newQuery = oldUri.getQuery();
if (newQuery == null) {
newQuery = appendQuery;
} else {
newQuery += "&" + appendQuery;
}
return new URI(
oldUri.getScheme(),
oldUri.getAuthority(),
oldUri.getPath(),
newQuery,
oldUri.getFragment())
.toString();
}
public String getDefaultImageName(final String imgUrl) {
String imageTitle = imgUrl.substring(imgUrl.lastIndexOf('/') + 1);
log.debug(String.format("Extracting name from url %s: %s", imgUrl, imageTitle));
return imgUrl;
}
@Getter
public static class Response {
private final Document document;
private final Header[] headers;
protected Response(Document document, Header[] headers) {
this.document = document;
this.headers = headers;
}
}
@Getter
public static class NameUrl {
private final String name;
private final String url;
public NameUrl(String name, String url) {
this.name = name;
this.url = url;
}
}
}
@@ -11,12 +11,16 @@ import tn.mnlr.vripper.exception.HtmlProcessorException;
@Service
public class HtmlProcessorService {
public Document clean(String htmlContent) throws HtmlProcessorException {
try {
TagNode clean = new HtmlCleaner().clean(htmlContent);
return new DomSerializer(new CleanerProperties()).createDOM(clean);
} catch (Exception e) {
throw new HtmlProcessorException(e);
private HtmlProcessorService() {
}
public Document clean(String htmlContent) throws HtmlProcessorException {
try {
TagNode clean = new HtmlCleaner().clean(htmlContent);
return new DomSerializer(new CleanerProperties()).createDOM(clean);
} catch (Exception e) {
throw new HtmlProcessorException(e);
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More