91 Commits

Author SHA1 Message Date
Sandeep Singh
96646c8f53
cloud templates targets sync (#2959)
* Add s3 bucket template provider

- Refactor the custom github template code
- add interface for template provider

* Validate if aws creds are passed if bucket flag

- refactor s3 provider struct to take client
- add function which returns the aws s3 client
- update error messages

* Add aws s3 bucket flags documentation in README.md

- Rename the github_test.go to customTemplate_test.go

* go mod update

* Move template provider code to pkg/external/customtemplates dir

* Added initial data_source sync to cloud

* Misc

* Add pagination to scan output and scan list (#2858)

* Add pagination to scan output and scan list

* Use time based parameters instead of page numbers

* Fix linting errors

* Do not check limits at client, check at server

* Remove unused constant

* Misc update

* Removed unnecessary flags

* Misc

* Misc

* Misc endpoint additions

* Added more routes

* Typo fix

* Misc fixes

* Misc

* Misc fixes to cloud target logic + use int for IDs

* Misc

* Misc fixes

* Misc

* Misc fixes

* readme update

* Add JSON output support for list-scan option (#2876)

* Add JSON output support for list-scan option

* Fix typo in cloud JSON output description

* Following changes

- Update status(finished, running) to be lower-case by default
- Convert status to upper-case in DisplayScanList()

* Update status to be lower-case by default

* Remove additional json flag, instead use existing

* Merge conflict

* Accomodate comment changes and restructure code

Co-authored-by: Jaideep K <jaideep@one2n.in>

* Use integer IDs for scan tasks

* Added get-templates-targets endpoint + JSON + validation

* Added target count list

* misc option / description updates

* Added changes as per code review

* duplicate options + typo updates

* Added tablewriter for tabular data writing by default

* Fixed list scan endpoint

* Review changes

* workflow fix

* Added cloud tags etc based filtering (#3070)

* Added omitempty for filtering request

* go mod tidy

* misc format update

Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
Co-authored-by: Ice3man <nizamulrana@gmail.com>
Co-authored-by: Jaideep Khandelwal <jdk2588@gmail.com>
Co-authored-by: Siddharth Shashikar <60960197+shashikarsiddharth@users.noreply.github.com>
Co-authored-by: Jaideep K <jaideep@one2n.in>
2022-12-21 22:48:43 +05:30
Shubham Rasal
d5a09e733a
Issue 2772 s3 provider support (#2825)
* Add s3 bucket template provider

- Refactor the custom github template code
- add interface for template provider

* Validate if aws creds are passed if bucket flag

- refactor s3 provider struct to take client
- add function which returns the aws s3 client
- update error messages

* Add aws s3 bucket flags documentation in README.md

- Rename the github_test.go to customTemplate_test.go

* go mod update

* Move template provider code to pkg/external/customtemplates dir

* Remove github and aws update variables from flag

* Rename CustomTemplateProvider to Provider

* Update integration and function command in makefile

* Update github test case, accept token

* readme update

* go mod tidy

* Update build-test.yml

* handle empty dir in s3

* Add requested changes

- download/update s3 and github only when `-ut` is passed
- only print the missing env variable for s3
- add the custom templates path in
  ~/.config/nuclei/.template-config.json

* print custom paths only if exists in config file

* misc update

* tag update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2022-12-02 03:57:00 +05:30
Shubham Rasal
ebfd2e648a
Remove nuclei-updatecheck-api as dependency (#2923)
* Remove nuclei-updatecheck-api as dependency

* Run go mod tidy

* go mod tidy

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-29 23:53:05 +05:30
vrenzolaverace
2aaf2a2158
Use utils helpers libraries (#2809) (#2810)
* Use utils helpers libraries (#2809)

* Use utils helpers libraries (#2809)
2022-11-07 01:54:23 +05:30
Shubham Rasal
721c4964d7
Issue 2613 custom template GitHub (#2630)
* Add custom template download/update support from github

- Accept the -gtr flag to accept the list of custom template
  repos(public/private)
- Accept the -gt flag for github token. It internally sets os.Env
  variable
- Update the flags from
   - -update to -nuclei-update for nuclei self update
   - -ut to -tup for template-update
   - -ud to -tud for custom template location
- Add github.go file which has code related to download and update
  custom templates repos.

* Reslove golint and test case error

* Take default template from community directory

- No need to give explicit community directory path.
- Update the integration test to support the change in path

* Update functional test script update template flag

* Update the path from community to nuclei-template

- Revert the code changes that were made to add community directory

* remove the comment

* Update the interactsh server url for testing

* Update race condition command

* update race condition cmd to download the templates

* Debug integration test failure

* update integration test to update templates

* Refactor downloadCustomTemplate function.

- Remove the log prining instead send the message.

* Add test case for custom template repo download

* move the download repo for loop into diff function

* refactor updateTemplate function.

* Create struct for github repos.

- Create customtemplate struct for repo.
- Add functions to customtemplate

* update readme.md file

* Refactor the downloadCustomTemplate function

- create const variables for github & community as template type
- Update gologger to INF
- Validate templateUpdate to accept only github & community value.
- Validate tempalteUpdate require githubTemplateRepo

* Resolve requested changes

* go mod update

* misc option update

* test update

* Revert back update-template flag to boolean.

- to update community templates
  `nuclei -ut`
- to update custom templates
  `nuclei -ut -gtr ehsandeep/mobile-nuclei-templates`

* Update readme to update flag documentation

* Update go.mod

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-03 20:27:18 +05:30
Mzack9999
30054d1fb6
Adding advanced template filtering (#2374)
* Adding advanced template filtering

* fixing bug in slice

* refactoring tests

* adding test cases

* increasing error verbosity

* fixing quoted fields with spaces

* adding more test cases

* fixing merge error

* fixing lint errors

* switching to []string

* updating tag filter tests

* updating functional tests

* fixing functional test cases

* updating syntax
2022-08-25 16:52:08 +05:30
Sajad
be73fde0db
use go-homedir instead of standard os.userhomedir (#2262)
* use go-homedir instead of standard os.userhomedir

* set r.templatesConfig before write attempt to avoid panic
2022-07-13 13:33:13 +05:30
Sami
ce79a8dc57
Template folder exists changes (#1825) 2022-04-11 11:59:22 +05:30
Mzack9999
0bab297328
improving default template path logic (#1806) 2022-04-05 04:03:49 +05:30
mzack
6e5e7f23e5 Implements github request with token fallback 2022-03-17 15:57:35 +01:00
sandeep
8096737e1a Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into update-custom-dir 2022-03-09 01:52:08 +05:30
mzack
64bea4da8c Merge branch 'dev' into issue-1594-regexp 2022-03-02 17:28:04 +01:00
mzack
630bf44c8b adding gh token 2022-03-02 17:02:51 +01:00
mzack
6e9ea761a7 Adding GH token detection 2022-03-02 16:56:32 +01:00
Sajad Parra
bdc1b337e0 fix setting custom nuclei-templates directory #1611 2022-03-02 16:50:20 +05:30
mzack
6746071979 Refactoring file templates to handle large files in chunks + removing deprecated io methods 2022-02-23 13:54:46 +01:00
Sajad
94d8c1d8e8
remove new-addtions file creation (#1624) 2022-02-22 13:10:57 +05:30
Ice3man
fea3fabdf2
Misc changes to update logic (#1212)
* Misc changes to update logic

* Misc adjustments to update logic

* update: build check

* update: revert test update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2021-12-16 13:59:19 +05:30
mzack9999
edc6e5f9b3 fix cross-os chunks calculation 2021-12-06 18:56:52 +01:00
Mzack9999
40d01412cb Fixing templates bundle unzip on windows 2021-12-06 18:12:39 +01:00
forgedhallpass
ec6889931d refactor: linter driven fixes
* x = x + ""  => x += ""
* pre-allocating slice with known size
* added t.Helper() methods in test helpers
* complex if-else conditions replaced by switches
* errors should be checked using error.Is() instead of ==
* function parameter should start with lower case letter
* removed unnecessary type definition
* variable/label naming convention: camelCase instead of snake_case
2021-11-25 17:57:22 +02:00
forgedhallpass
fdd22ab668 refactor: Wrap errors using %w in fmt.Errorf
see:
 * https://github.com/xxpxxxxp/intellij-plugin-golangci-lint/blob/master/explanation/goerr113.md
 * https://go.dev/blog/go1.13-errors#wrapping-errors-with-w
2021-11-25 16:24:37 +02:00
forgedhallpass
f9c214a66f refactor: update logic refactor to make it more testable
Introduced logic to test for zip slip (path traversal)
2021-11-25 16:24:28 +02:00
forgedhallpass
4bccb6cf8a fix: File and directory creation permission changes
Directories: 0755 (5 - group and other can read cd into the directory and read it's content)
Files: 0644 (4 - group and other can only read the created files)
Tests files: 0777
2021-11-25 14:37:57 +02:00
forgedhallpass
f96168fc2c [feature] Add coloring to debug information #999
* Reverted the bug introduced by mistake (checking file exists using os.IsExist vs !os.IsNotExist)
2021-10-08 20:17:37 +03:00
forgedhallpass
0a9704ebd8 [feature] Add coloring to debug information #999
* code-review fixes
2021-10-07 20:54:12 +03:00
forgedhallpass
8fdc9a7164 Minor optimization: short-circuit the updateTemplates logic if template update was not requested 2021-09-29 19:44:37 +03:00
forgedhallpass
2baf695709 Merge remote-tracking branch 'origin/dev' into code_smells 2021-09-16 20:00:21 +03:00
sandeep
cefbd376ef misc update 2021-09-16 17:27:06 +05:30
Ice3man543
9bbcb57be0 Added disclaimer for auto update to comments 2021-09-16 16:49:09 +05:30
Ice3man543
4d52fb45e4 Misc changes to update and ignore handling 2021-09-15 04:01:40 +05:30
Ice3man543
6667f285d3 fix: consider missing templates directory in update check 2021-09-15 03:48:40 +05:30
Ice3man543
0f232bce25 Don't show message without -ut 2021-09-13 15:47:29 +05:30
Ice3man543
04a4159fa5 Misc changes to update mechanism 2021-09-13 15:45:24 +05:30
Ice3man543
be46cb2b32 Show not found new templates message 2021-09-13 15:32:40 +05:30
Ice3man543
638c7633cb Misc updates 2021-09-13 15:03:04 +05:30
Ice3man543
8a1a739355 Misc 2021-09-13 14:57:30 +05:30
Ice3man543
5cda839152 Fixed an issue with nuclei version for templates 2021-09-13 14:38:22 +05:30
Ice3man543
cb39fd9d13 Use separate nuclei version check infra 2021-09-11 18:40:07 +05:30
forgedhallpass
0ce33927c7 Typo fixes. 2021-09-07 17:31:46 +03:00
forgedhallpass
8b8c16dd8e Improve readability of some methods 2021-09-01 17:36:07 +03:00
forgedhallpass
f9eb8ba8ac Changed/removed some documentation/comments 2021-09-01 17:34:51 +03:00
forgedhallpass
40d88d2304 In-lined error checks, reduced scope of error variables, introduced new error variables instead of re-using them 2021-08-31 12:55:52 +03:00
Ice3man543
fb2ec8f5d2 Fixed problems with nuclei templates update logic 2021-08-25 18:52:51 +05:30
Ice3man543
0088893310 Fix for template directory location override 2021-08-25 18:24:10 +05:30
forgedhallpass
07ca97fdaf Efforts to make working with files OS-agnostic.
Replacing "path." methods to "filepath." in order to make the code OS independent.
2021-08-23 16:42:29 +03:00
forgedhallpass
71c0d217e2 Replacing "path." methods to "filepath." in order to make the code OS independent 2021-08-23 14:54:28 +03:00
Ice3man543
5b03d25743 Do no check for template updates with flag 2021-07-29 19:24:39 +05:30
Ice3man543
20e519ec04 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into dev 2021-07-25 16:06:35 +05:30
Ice3man543
8650fdbe5a Show message on no new update 2021-07-25 16:06:12 +05:30