e1d3f474a4
* multi proto request genesis * adds template context dynamic vars * feat: proto level resp variables * remove proto prefix hacky logic * implement template ctx args * remove old var name logic * improve AddTemplateVars func * add multi proto comments+docs * vardump with sorted keys * fix race condition in ctx args * default initialize ctx args * use generic map * index variables with multiple values * fix nil cookies * use synclock map * fix build failure * fix lint error * resolve merge conflicts * multi proto: add unit+ integration tests * fix unit tests * Issue 3339 headless fuzz (#3790) * Basic headless fuzzing * Remove debug statements * Add integration tests * Update template * Fix recognize payload value in matcher * Update tempalte * use req.SetURL() --------- Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> * Auto Generate Syntax Docs + JSONSchema [Fri Jun 9 00:23:32 UTC 2023] 🤖 * Add headless header and status matchers (#3794) * add headless header and status matchers * rename headers as header * add integration test for header+status * fix typo --------- Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
multi protocol execution
Implementation
when template is unmarshalled, if it uses more than one protocol, it will be converted to a multi protocol and the order of the protocols will be preserved as they were in the template and are stored in Request.Queue when template is compiled , we iterate over queue and compile all the requests in the queue
Execution
when multi protocol template is executed , all protocol requests present in Queue are executed in order and dynamic values extracted are added to template context.
- Protocol Responses
apart from extracted
internal:truevalues response fields/values of protocol are added to template context atExecutorOptions.TemplateCtxwhich takes care of sync and other issues if any. all response fields are prefixed with template type prefix ex:ssl_subject_dn
Other Methods
Such templates are usually used when a particular vulnerability requires more than one protocol to be executed and in such cases the final result is core of the logic hence all methods such as Ex: MakeResultEventItem, MakeResultEvent, GetCompiledOperators are not implemented in multi protocol and just call the same method on last protocol in queue
Adding New Protocol to multi protocol execution logic
while logic/implementation of multi protocol execution is abstracted. it requires 3 statements to be added in newly implemented protocol to make response fields of that protocol available to global context
- Add
request.options.TemplateCtx.GetAll()to variablesMap inExecuteWithResultsMethod just aboverequest.options.Variables.Evaluate
// example
values := generators.MergeMaps(payloadValues, hostnameVariables, request.options.TemplateCtx.GetAll())
variablesMap := request.options.Variables.Evaluate(values)
- Add all response fields to template context just after response map is available
outputEvent := request.responseToDSLMap(compiledRequest, response, domain, question, traceData)
// expose response variables in proto_var format
// this is no-op if the template is not a multi protocol template
request.options.AddTemplateVars(request.Type(), outputEvent)
- Append all available template context values to outputEvent
// add variables from template context before matching/extraction
outputEvent = generators.MergeMaps(outputEvent, request.options.TemplateCtx.GetAll())
adding these 3 statements takes care of all logic related to multi protocol execution
Exceptions
- statements 1 & 2 are intentionally skipped in
fileprotocol to avoid redundant data- file/dir input paths don't contain variables or are used in path (yet)
- since files are processed by scanning each line. adding statement 2 will unintenionally load all file(s) data