Ramana Reddy 6707bc777a
fix showing multiple failure matches per template on -ms set (#3770)
* fix showing multiple failure matchers per template
add integration test

* exclude AS134029 from unit test

* Add flag for match status per request

* chore(deps): bump golangci/golangci-lint-action from 3.4.0 to 3.5.0 (#3777)

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.4.0...v3.5.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/xanzy/go-gitlab in /v2 (#3778)

Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.83.0 to 0.84.0.
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](https://github.com/xanzy/go-gitlab/compare/v0.83.0...v0.84.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/spf13/cast from 1.5.0 to 1.5.1 in /v2 (#3780)

Bumps [github.com/spf13/cast](https://github.com/spf13/cast) from 1.5.0 to 1.5.1.
- [Release notes](https://github.com/spf13/cast/releases)
- [Commits](https://github.com/spf13/cast/compare/v1.5.0...v1.5.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cast
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* enable no-httpx when passive scan is launched (#3789)

* chore(deps): bump github.com/projectdiscovery/fastdialer from 0.0.26 to 0.0.28 in /v2 (#3779)

* chore(deps): bump github.com/projectdiscovery/fastdialer in /v2

Bumps [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) from 0.0.26 to 0.0.28.
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.0.26...v0.0.28)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump retryabledns to 0.28

* Update the retryabledns

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>

* deprecatedProtocolNameTemplates concurrent map writes (#3785)

* deprecatedProtocolNameTemplates

* use syncLock

* fix lint error

* change version in deprecated warning msg

* comment asnmap expand unit test

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

* chore(deps): bump golang from 1.20.4-alpine to 1.20.5-alpine (#3809)

Bumps golang from 1.20.4-alpine to 1.20.5-alpine.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/go-playground/validator/v10 in /v2 (#3810)

Bumps [github.com/go-playground/validator/v10](https://github.com/go-playground/validator) from 10.11.2 to 10.14.1.
- [Release notes](https://github.com/go-playground/validator/releases)
- [Commits](https://github.com/go-playground/validator/compare/v10.11.2...v10.14.1)

---
updated-dependencies:
- dependency-name: github.com/go-playground/validator/v10
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/projectdiscovery/rawhttp in /v2 (#3811)

Bumps [github.com/projectdiscovery/rawhttp](https://github.com/projectdiscovery/rawhttp) from 0.1.11 to 0.1.13.
- [Release notes](https://github.com/projectdiscovery/rawhttp/releases)
- [Commits](https://github.com/projectdiscovery/rawhttp/compare/v0.1.11...v0.1.13)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/rawhttp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/go-git/go-git/v5 from 5.6.1 to 5.7.0 in /v2 (#3812)

Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) from 5.6.1 to 5.7.0.
- [Release notes](https://github.com/go-git/go-git/releases)
- [Commits](https://github.com/go-git/go-git/compare/v5.6.1...v5.7.0)

---
updated-dependencies:
- dependency-name: github.com/go-git/go-git/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/projectdiscovery/hmap in /v2 (#3781)

Bumps [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) from 0.0.11 to 0.0.13.
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.11...v0.0.13)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/hmap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Using safe dereferencing

* adding comment

* fixing and condition

* fixing test id

* adding integration test

* update goflags dependency

* update goflags dependency

* bump goflags v0.1.9 => v0.1.10

* handle failure matcher flags logic at executor itself

* add integration test to matcher status per request

* Adding random tls impersonate (#3844)

* adding random tls impersonate

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>

* Use templateman enhance api to populate CVE info (#3788)

* use templateman enhance api to populate cve info

* rename cve-annotate => tmc
add additional flags to format, lint and enhance template using templateman apis

* minior changes

* remove duplicate code

* misc update

* Add validate and error log option

* print if updated

* print format and enhance only if updated

* make max-request optional

* fix reference unmarshal error

* fix removing self-contained tag

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>

* fix matcher status with network protocol

* fix test

* remove -msr flag

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
Co-authored-by: 三米前有蕉皮 <kali-team@qq.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2023-06-30 23:32:00 +05:30

186 lines
6.0 KiB
Go

package executer
import (
"fmt"
"strings"
"sync/atomic"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/writer"
)
// Executer executes a group of requests for a protocol
type Executer struct {
requests []protocols.Request
options *protocols.ExecutorOptions
}
var _ protocols.Executer = &Executer{}
// NewExecuter creates a new request executer for list of requests
func NewExecuter(requests []protocols.Request, options *protocols.ExecutorOptions) *Executer {
return &Executer{requests: requests, options: options}
}
// Compile compiles the execution generators preparing any requests possible.
func (e *Executer) Compile() error {
cliOptions := e.options.Options
for _, request := range e.requests {
if err := request.Compile(e.options); err != nil {
var dslCompilationError *dsl.CompilationError
if errors.As(err, &dslCompilationError) {
if cliOptions.Verbose {
rawErrorMessage := dslCompilationError.Error()
formattedErrorMessage := strings.ToUpper(rawErrorMessage[:1]) + rawErrorMessage[1:] + "."
gologger.Warning().Msgf(formattedErrorMessage)
gologger.Info().Msgf("The available custom DSL functions are:")
fmt.Println(dsl.GetPrintableDslFunctionSignatures(cliOptions.NoColor))
}
}
return err
}
}
return nil
}
// Requests returns the total number of requests the rule will perform
func (e *Executer) Requests() int {
var count int
for _, request := range e.requests {
count += request.Requests()
}
return count
}
// Execute executes the protocol group and returns true or false if results were found.
func (e *Executer) Execute(input *contextargs.Context) (bool, error) {
results := &atomic.Bool{}
dynamicValues := make(map[string]interface{})
if input.HasArgs() {
input.ForEach(func(key string, value interface{}) {
dynamicValues[key] = value
})
}
previous := make(map[string]interface{})
var lastMatcherEvent *output.InternalWrappedEvent
writeFailureCallback := func(event *output.InternalWrappedEvent, matcherStatus bool) {
if !results.Load() && matcherStatus {
if err := e.options.Output.WriteFailure(event.InternalEvent); err != nil {
gologger.Warning().Msgf("Could not write failure event to output: %s\n", err)
}
results.CompareAndSwap(false, true)
}
}
for _, req := range e.requests {
inputItem := input.Clone()
if e.options.InputHelper != nil && input.MetaInput.Input != "" {
if inputItem.MetaInput.Input = e.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
return false, nil
}
}
err := req.ExecuteWithResults(inputItem, dynamicValues, previous, func(event *output.InternalWrappedEvent) {
ID := req.GetID()
if ID != "" {
builder := &strings.Builder{}
for k, v := range event.InternalEvent {
builder.WriteString(ID)
builder.WriteString("_")
builder.WriteString(k)
previous[builder.String()] = v
builder.Reset()
}
}
// If no results were found, and also interactsh is not being used
// in that case we can skip it, otherwise we've to show failure in
// case of matcher-status flag.
if !event.HasOperatorResult() && !event.UsesInteractsh {
lastMatcherEvent = event
} else {
if writer.WriteResult(event, e.options.Output, e.options.Progress, e.options.IssuesClient) {
results.CompareAndSwap(false, true)
} else {
lastMatcherEvent = event
}
}
})
if err != nil {
if e.options.HostErrorsCache != nil {
e.options.HostErrorsCache.MarkFailed(input.MetaInput.ID(), err)
}
gologger.Warning().Msgf("[%s] Could not execute request for %s: %s\n", e.options.TemplateID, input.MetaInput.PrettyPrint(), err)
}
// If a match was found and stop at first match is set, break out of the loop and return
if results.Load() && (e.options.StopAtFirstMatch || e.options.Options.StopAtFirstMatch) {
break
}
}
if lastMatcherEvent != nil {
writeFailureCallback(lastMatcherEvent, e.options.Options.MatcherStatus)
}
return results.Load(), nil
}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (e *Executer) ExecuteWithResults(input *contextargs.Context, callback protocols.OutputEventCallback) error {
dynamicValues := make(map[string]interface{})
if input.HasArgs() {
input.ForEach(func(key string, value interface{}) {
dynamicValues[key] = value
})
}
previous := make(map[string]interface{})
results := &atomic.Bool{}
for _, req := range e.requests {
req := req
inputItem := input.Clone()
if e.options.InputHelper != nil && input.MetaInput.Input != "" {
if inputItem.MetaInput.Input = e.options.InputHelper.Transform(input.MetaInput.Input, req.Type()); input.MetaInput.Input == "" {
return nil
}
}
err := req.ExecuteWithResults(inputItem, dynamicValues, previous, func(event *output.InternalWrappedEvent) {
ID := req.GetID()
if ID != "" {
builder := &strings.Builder{}
for k, v := range event.InternalEvent {
builder.WriteString(ID)
builder.WriteString("_")
builder.WriteString(k)
previous[builder.String()] = v
builder.Reset()
}
}
if event.OperatorsResult == nil {
return
}
results.CompareAndSwap(false, true)
callback(event)
})
if err != nil {
if e.options.HostErrorsCache != nil {
e.options.HostErrorsCache.MarkFailed(input.MetaInput.ID(), err)
}
gologger.Warning().Msgf("[%s] Could not execute request for %s: %s\n", e.options.TemplateID, input.MetaInput.PrettyPrint(), err)
}
// If a match was found and stop at first match is set, break out of the loop and return
if results.Load() && (e.options.StopAtFirstMatch || e.options.Options.StopAtFirstMatch) {
break
}
}
return nil
}