359 Commits

Author SHA1 Message Date
Jaideep Khandelwal
a1642be911
Enable/Disable a reporting source (#3183)
* Enable/Disable a reporting source

* misc options update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-01-15 22:21:16 +05:30
Ice3man
67c094444e
Added cloud scan progress tracking using stats (#3180)
* Added cloud scan progress tracking using stats

* Changed log messsage

* Fixed linting error

* Fixed bug in progress calculation logic

* Changed requests to input with cloud flag

* Changed progress name + removed redundant fields
2023-01-13 13:41:05 +05:30
Shubham Rasal
25fcae1493
add reporting source in nuclei-cloud (#3151)
* add reporting source in nuclei-cloud

- `nuclei -cloud -rc reporting-config.yaml`

* update error message

* add severity options for jira,(used for cloud only)
2023-01-10 22:49:01 +05:30
Ice3man
ded218a88a Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into cloud-bug-fixes 2022-12-28 20:43:12 +05:30
Mzack9999
260dd1a2c4
Disable include directive preprocessing by default (#3045)
* adding strict syntax check

* returning error on disabled preprocessing

* adding check on matchers
2022-12-28 19:16:34 +05:30
Tarun Koyalwar
ff17d12ced
adds scan strategy (#3075) 2022-12-28 02:18:00 +05:30
Mzack9999
34976029d3
removing most go routine leaks (#3073)
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-12-24 19:22:14 +05:30
Ice3man
3411293e6a Cloud related REST API bug fixes 2022-12-23 11:08:38 +05:30
Sandeep Singh
96646c8f53
cloud templates targets sync (#2959)
* Add s3 bucket template provider

- Refactor the custom github template code
- add interface for template provider

* Validate if aws creds are passed if bucket flag

- refactor s3 provider struct to take client
- add function which returns the aws s3 client
- update error messages

* Add aws s3 bucket flags documentation in README.md

- Rename the github_test.go to customTemplate_test.go

* go mod update

* Move template provider code to pkg/external/customtemplates dir

* Added initial data_source sync to cloud

* Misc

* Add pagination to scan output and scan list (#2858)

* Add pagination to scan output and scan list

* Use time based parameters instead of page numbers

* Fix linting errors

* Do not check limits at client, check at server

* Remove unused constant

* Misc update

* Removed unnecessary flags

* Misc

* Misc

* Misc endpoint additions

* Added more routes

* Typo fix

* Misc fixes

* Misc

* Misc fixes to cloud target logic + use int for IDs

* Misc

* Misc fixes

* Misc

* Misc fixes

* readme update

* Add JSON output support for list-scan option (#2876)

* Add JSON output support for list-scan option

* Fix typo in cloud JSON output description

* Following changes

- Update status(finished, running) to be lower-case by default
- Convert status to upper-case in DisplayScanList()

* Update status to be lower-case by default

* Remove additional json flag, instead use existing

* Merge conflict

* Accomodate comment changes and restructure code

Co-authored-by: Jaideep K <jaideep@one2n.in>

* Use integer IDs for scan tasks

* Added get-templates-targets endpoint + JSON + validation

* Added target count list

* misc option / description updates

* Added changes as per code review

* duplicate options + typo updates

* Added tablewriter for tabular data writing by default

* Fixed list scan endpoint

* Review changes

* workflow fix

* Added cloud tags etc based filtering (#3070)

* Added omitempty for filtering request

* go mod tidy

* misc format update

Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
Co-authored-by: Ice3man <nizamulrana@gmail.com>
Co-authored-by: Jaideep Khandelwal <jdk2588@gmail.com>
Co-authored-by: Siddharth Shashikar <60960197+shashikarsiddharth@users.noreply.github.com>
Co-authored-by: Jaideep K <jaideep@one2n.in>
2022-12-21 22:48:43 +05:30
Ice3man
bfa0bd7bee Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into cloud-templates-targets-sync 2022-12-17 22:52:39 +05:30
Ice3man
3409f9fca3 Added changes as per code review 2022-12-16 23:10:43 +05:30
xm1k3
0e3be82c72
Resuming rewriting the output file contents, so previous data removed (#2890)
* used OpenFile instead of Create()

* reverted to original mode

* fixes and resume flag added

* fix on noTimestapt var

* fix on flag

* better code refactoring

* fix on debug error

* code refactoring on file management
2022-12-15 21:11:23 +05:30
Ice3man
a6c88817fa Added get-templates-targets endpoint + JSON + validation 2022-12-12 23:39:21 +05:30
Ice3man
2a1721a162 Use integer IDs for scan tasks 2022-12-12 20:31:15 +05:30
Ice3man
d42f0caecb Misc fixes 2022-12-09 14:55:51 +05:30
Ice3man
1f8bbe5ed2 Misc fixes to cloud target logic + use int for IDs 2022-12-09 00:15:18 +05:30
Ice3man
50e766a19d Misc 2022-12-08 20:45:39 +05:30
Ice3man
a3e3c1cf3d Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into cloud-templates-targets-sync 2022-12-08 20:31:23 +05:30
Ice3man
3181d1fa2d Added more routes 2022-12-06 14:11:32 +05:30
Ice3man
5a1a427afc Misc endpoint additions 2022-12-05 23:02:18 +05:30
Ice3man
514c6e2d1e
Added timestamp optional flag + user-agent to probing (#2962)
* Added timestamp optional flag + user-agent to probing

* fix typo

* misc update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-12-04 22:16:55 +05:30
Ice3man
ae36dec68c Misc 2022-12-04 21:48:05 +05:30
Ice3man
b52254bb74 Misc 2022-12-04 20:43:19 +05:30
Shubham Rasal
d5a09e733a
Issue 2772 s3 provider support (#2825)
* Add s3 bucket template provider

- Refactor the custom github template code
- add interface for template provider

* Validate if aws creds are passed if bucket flag

- refactor s3 provider struct to take client
- add function which returns the aws s3 client
- update error messages

* Add aws s3 bucket flags documentation in README.md

- Rename the github_test.go to customTemplate_test.go

* go mod update

* Move template provider code to pkg/external/customtemplates dir

* Remove github and aws update variables from flag

* Rename CustomTemplateProvider to Provider

* Update integration and function command in makefile

* Update github test case, accept token

* readme update

* go mod tidy

* Update build-test.yml

* handle empty dir in s3

* Add requested changes

- download/update s3 and github only when `-ut` is passed
- only print the missing env variable for s3
- add the custom templates path in
  ~/.config/nuclei/.template-config.json

* print custom paths only if exists in config file

* misc update

* tag update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2022-12-02 03:57:00 +05:30
Víctor
9c17284616
Display tpl contents (#2906)
* New "td" flag, displays the highlighted template contents
New verboseTemplate method to avoid duplicate code
Grouped (and sorted) template list per directory

* Updated README about the td flag

* Going back to the previous template list format

The new one can't be pipelined

* Implicit template list on template display
Respect --no-color option to disable colors when -td is used

* misc option update

Co-authored-by: Víctor Zamanillo <victor.zamanillo@cifraeducacion.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-29 17:48:44 +05:30
dependabot[bot]
e0dfc476c3
chore(deps): bump github.com/projectdiscovery/ratelimit from 0.0.1 to 0.0.2 in /v2 (#2915)
* chore(deps): bump github.com/projectdiscovery/ratelimit in /v2

Bumps [github.com/projectdiscovery/ratelimit](https://github.com/projectdiscovery/ratelimit) from 0.0.1 to 0.0.2.
- [Release notes](https://github.com/projectdiscovery/ratelimit/releases)
- [Commits](https://github.com/projectdiscovery/ratelimit/compare/v0.0.1...v0.0.2)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/ratelimit
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* fixing int type

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2022-11-28 12:49:30 +05:30
Ice3man
694b0f3ea1 Merge branch 'cloud-templates-targets-sync' of https://github.com/projectdiscovery/nuclei into cloud-templates-targets-sync 2022-11-24 14:00:43 +05:30
Jaideep Khandelwal
2d859a41fc
Add pagination to scan output and scan list (#2858)
* Add pagination to scan output and scan list

* Use time based parameters instead of page numbers

* Fix linting errors

* Do not check limits at client, check at server

* Remove unused constant
2022-11-24 14:00:22 +05:30
Ice3man
cbbbe1b5b7 Merge branch 'issue-2772-s3-provider-support' of https://github.com/projectdiscovery/nuclei into cloud-templates-targets-sync 2022-11-24 14:00:17 +05:30
Shubham Rasal
6b142d794a
Issue 2254 uncover integration (#2786)
* nuclei -uq 'vuln:CVE-2021-26855' -t cves/2021/

- `nuclei -uq 'vuln:CVE-2021-26855' -t cves/2021/`

* Add automatic template execution using metadata

- Query uncover after the template is loaded.
- Add the received hosts to the input provider from uncover
- Make NormalizeStoreInputValue() function public to add hosts from the
  runner after uncover hosts received.

* run go mod tidy

* Remove unnecessary comments

* Resolve the requested changes

- move uncover code to protocols/common/uncover package
- Use uncover delay to create uncover rate limiter
- Use single ratelimiter object and remove not required ratelimiters
- Create Set() method for input provider interface
- Rename normalizeStoreInputValue to Set() method

* Solved the uncover running twice.

- flag StringSliceVarP adds the default value twice in the variable
- Check if provider keys exists or not
- Add uncover help block to english readme.md

* Add uncover field functionality

- ./nuclei -uq 'vuln:CVE-2021-26855' -t dns -duc -uf host
- ./nuclei -uq 'vuln:CVE-2021-26855' -t dns -duc -uf ip:port

* Update error messages and solve nuclei hang for wrong uncover engine

- Get uncover engine values from uncover package

* Resolve merge conflicts

* misc option update

* Update logging for templates

- remove duplicate env log printing
- Log message for template queries

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-16 11:12:39 +05:30
Ice3man
6d2d75ac8f Added initial data_source sync to cloud 2022-11-15 01:37:00 +05:30
Sandeep Singh
0f567cd95c
Merge branch 'dev' into issue-2772-s3-provider-support 2022-11-12 19:42:30 +05:30
vrenzolaverace
2403c50c36
Add targets number in output banner (#2798)
* Add targets number in output banner (#2797)

* Add targets number in output banner (#2797)
2022-11-12 18:05:49 +05:30
shubhamrasal
97c5922ec6 Merge branch 'dev' into issue-2772-s3-provider-support 2022-11-10 17:39:27 +05:30
shubhamrasal
fb36d89f3c Move template provider code to pkg/external/customtemplates dir 2022-11-10 17:19:06 +05:30
Mzack9999
1fbbca66f9
Adding support to scan all v4/v6 IPs (#2709)
* Adding support to scan all v4/v6 IPs

* adding tests

* metainput prototype

* using new signature

* fixing nil pointer

* adding request context with metadata

* removing log instruction

* fixing merge conflicts

* adding clone helpers

* attempting to fix ipv6 square parenthesis wrap

* fixing dialed ip info

* fixing syntax

* fixing output ip selection

* adding integration tests

* disabling test due to gh ipv6 issue

* using ipv4 only due to GH limited networking

* extending metainput marshaling

* fixing hmap key

* adding test for httpx integration

* fixing lint error

* reworking marshaling/id-calculation

* adding ip version validation

* improving handling non url targets

* fixing condition check
2022-11-09 18:48:56 +05:30
shubhamrasal
146e328c6b Add s3 bucket template provider
- Refactor the custom github template code
- add interface for template provider
2022-11-07 17:14:02 +05:30
vrenzolaverace
2aaf2a2158
Use utils helpers libraries (#2809) (#2810)
* Use utils helpers libraries (#2809)

* Use utils helpers libraries (#2809)
2022-11-07 01:54:23 +05:30
Shubham Rasal
721c4964d7
Issue 2613 custom template GitHub (#2630)
* Add custom template download/update support from github

- Accept the -gtr flag to accept the list of custom template
  repos(public/private)
- Accept the -gt flag for github token. It internally sets os.Env
  variable
- Update the flags from
   - -update to -nuclei-update for nuclei self update
   - -ut to -tup for template-update
   - -ud to -tud for custom template location
- Add github.go file which has code related to download and update
  custom templates repos.

* Reslove golint and test case error

* Take default template from community directory

- No need to give explicit community directory path.
- Update the integration test to support the change in path

* Update functional test script update template flag

* Update the path from community to nuclei-template

- Revert the code changes that were made to add community directory

* remove the comment

* Update the interactsh server url for testing

* Update race condition command

* update race condition cmd to download the templates

* Debug integration test failure

* update integration test to update templates

* Refactor downloadCustomTemplate function.

- Remove the log prining instead send the message.

* Add test case for custom template repo download

* move the download repo for loop into diff function

* refactor updateTemplate function.

* Create struct for github repos.

- Create customtemplate struct for repo.
- Add functions to customtemplate

* update readme.md file

* Refactor the downloadCustomTemplate function

- create const variables for github & community as template type
- Update gologger to INF
- Validate templateUpdate to accept only github & community value.
- Validate tempalteUpdate require githubTemplateRepo

* Resolve requested changes

* go mod update

* misc option update

* test update

* Revert back update-template flag to boolean.

- to update community templates
  `nuclei -ut`
- to update custom templates
  `nuclei -ut -gtr ehsandeep/mobile-nuclei-templates`

* Update readme to update flag documentation

* Update go.mod

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-03 20:27:18 +05:30
Jaideep Khandelwal
4cfde111f4
Feature 18 cloud flags (#2708)
* Add cloud flags for nuclei.

* Add flag to get output for a particular scan ID

* Add some comments to the function.

* Get timestamp and id for scan list

* Fix linting errors

* Check if type is enumeration.

* Do not show deleted scans.

* Do not use filter_result, create client once and use it everywhere with
runner.

* Fix the output of scan list to be better

* Format the nuclei scan output list.

* Remove unused constant

* misc option update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-10-22 04:06:52 +05:30
Ice3man
363ffb75db
Added probing for URL + input based on protocol (#2614)
* Added workflow names based condition

* Added conditional filtering to workflow executor

* Replaced names with single name stringslice

* Added probing for URL + input based on protocol

* Remove debug comments

* Fixed typo

* Fixed failing tests

* Fixed workflow matcher condition + tests

* Fixed workflow item name

* Switch to if-else

* Fixed review comment strict

* Increase bulk size

* Added default port for SSL protocol + misc changes

* Fixed failing tests

* Fixed misc changes to executer

* Fixed failing self-contained and offlinehttp tests

* Fixed atomic increment operation

* misc update

* Fixed failing builds

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-10-20 17:23:00 +05:30
LuitelSamikshya
944d24a252 mics changes 2022-10-13 09:30:25 -05:00
LuitelSamikshya
cb0da81a14 ratelimit library 2022-10-12 22:04:37 -05:00
Ice3man
09ceb29ba3 Fixed build error on 32bit arch 2022-10-08 01:55:18 +05:30
Ice3man
fc27fc94a5
Added default config generation for reporting options (#2605) 2022-09-27 02:40:34 +05:30
Sami
512eab7b21
tl flag improvements (#2596)
* tl flag improvements

* tl flag enhancement with additional filters

* added ExcluedTags filter

* tl flas to list template paths

* using stdout

* misc update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-09-21 03:03:57 +05:30
Mzack9999
99c14f4c9c
implementation of rate limiter with bucket refill and unrestricted token burst (#2536)
* implementation of rate limiter with interval burst

* fixing import path

* fixing syntax

* adding tests

* fixing lint errors

* adding support for context

* moving rate limiter earlier to avoid hitting timeout
2022-09-19 17:09:28 +05:30
mzack
37c016a5f0 fixing lint errors 2022-09-19 08:38:52 +02:00
Ice3man
04b47b0309
Added custom json-unmarshaller + misc updates (#2556)
* Added custom json-unmarshaller + misc updates

* Added support for nuclei-cloud based scan execution

* Removed unnecessary files

* Misc

* Changes as per review comments

* misc option update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-09-19 01:13:59 +05:30
Mzack9999
30054d1fb6
Adding advanced template filtering (#2374)
* Adding advanced template filtering

* fixing bug in slice

* refactoring tests

* adding test cases

* increasing error verbosity

* fixing quoted fields with spaces

* adding more test cases

* fixing merge error

* fixing lint errors

* switching to []string

* updating tag filter tests

* updating functional tests

* fixing functional test cases

* updating syntax
2022-08-25 16:52:08 +05:30