Skip to content

dhondta/webgrep

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

55 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WebGrep Tweet

Grep Web pages and their resources.

PyPi Platform Read The Docs Known Vulnerabilities License

This self-contained tool relies on the well-known grep tool for grepping Web pages. It binds nearly every option of the original tool and also provides additional features like deobfuscating Javascript or appyling OCR on images before grepping downloaded resources.

$ pip install webgrep-tool 

⏩ Quick Start

  1. Help
$ webgrep --help usage: webgrep [OPTION]... PATTERN [URL]... Search for PATTERN in each input URL and its related resources (images, scripts and style sheets). By default, - resources are NOT downloaded - response HTTP headers are NOT included in grepping ; use '--include-headers' - PATTERN is a basic regular expression (BRE) ; use '-E' for extended (ERE) Important note: webgrep does not handle recursion (in other words, it does not spider additional web pages). Examples: webgrep example http://www.example.com # will only grep on HTML code webgrep -r example http://www.example.com # will only grep on LOCAL images, ... webgrep -R example http://www.example.com # will only grep on ALL images, ... Regexp selection and interpretation: -e REGEXP, --regexp REGEXP use PATTERN for matching -f FILE, --file FILE obtain PATTERN from FILE -E, --extended-regexp PATTERN is an extended regular expression (ERE) -F, --fixed-strings PATTERN is a set of newline-separated fixed strings -G, --basic-regexp PATTERN is a basic regular expression (BRE) -P, --perl-regexp PATTERN is a Perl regular expression -i, --ignore-case ignore case distinctions -w, --word-regexp force PATTERN to match only whole words -x, --line-regexp force PATTERN to match only whole lines -z, --null-data a data line ends in 0 byte, not newline Miscellaneous: -s, --no-messages suppress error messages -v, --invert-match select non-matching lines -V, --version print version information and exit --help display this help and exit --verbose verbose mode --keep-files keep temporary files in the temporary directory --temp-dir TMP define the temporary directory (default: /tmp/webgrep) Output control: -m NUM, --max-count NUM stop after NUM matches -b, --byte-offset print the byte offset with output lines -n, --line-number print line number with output lines --line-buffered flush output on every line -H, --with-filename print the file name for each match -h, --no-filename suppress the file name prefix on output --label LABEL use LABEL as the standard input filename prefix -o, --only-matching show only the part of a line matching PATTERN -q, --quiet, --silent suppress all normal output --binary-files TYPE assume that binary files are TYPE; TYPE is 'binary', 'text', or 'without-match' -a, --text equivalent to --binary-files=text -I equivalent to --binary-files=without-match -L, --files-without-match print only names of FILEs containing no match -l, --files-with-match print only names of FILEs containing matches -c, --count print only a count of matching lines per FILE -T, --initial-tab make tabs line up (if needed) -Z, --null print 0 byte after FILE name Context control: -B NUM, --before-context NUM print NUM lines of leading context -A NUM, --after-context NUM print NUM lines of trailing context -C NUM, --context NUM print NUM lines of output context Web options: -r, --local-resources also grep local resources (same-origin) -R, --all-resources also grep all resources (even non-same-origin) --include-headers also grep HTTP headers --cookie COOKIE use a session cookie in the HTTP headers --referer REFERER provide the referer in the HTTP headers Proxy settings (by default, system proxy settings are used): -d, --disable-proxy manually disable proxy --http-proxy HTTP manually set the HTTP proxy --https-proxy HTTPS manually set the HTTPS proxy Please report bugs on GitHub: https://github.com/dhondta/webgrep 
  1. Example
$ ./webgrep -R Welcome https://github.com Welcome home, <br>developers 

📌 Resource Handlers

Definitions:

  • Resource (what is being processed): Web page, images, Javascript, CSS
  • Handler (how a resource is processed): CSS unminifying, OCR, deobfuscation, EXIF data retrieval, ...

The handlers are defined in the # --...-- HANDLERS SECTION --...-- of the code. Currently available handlers :

  1. Images
  • EXIF: using exiftool
  • Steganography: using steghide (with a blank password)
  • Strings: using strings
  • OCR: using tesseract
  1. Scripts
  • Javascript beautifying and deobfuscation: using jsbeautifier
  1. Styles
  • Unminifying: using regular expressions

Note: images found in the CSS files are also processed.

👏 Supporters

Stargazers repo roster for @dhondta/webgrep

Forkers repo roster for @dhondta/webgrep

Back to top

About

Grep Web pages with extra features like JS deobfuscation and OCR

Topics

Resources

License

Stars

Watchers

Forks

Languages