SN0INT(1)

NAME
sn0int - OSINT framework and package manager

todo

RUNNING YOUR FIRST INVESTIGATION

This page is going to guide you through the process of setting up your environment and running your first investigation.

  Installing the default modules

By default, sn0int doesn't have any modules installed. If you start up sn0int it's going to download some files that it needs and then suggests to install a number of recommended modules:

$ sn0int
                   ___/           .
     ____ , __   .'  /\ ` , __   _/_
    (     |'  `. |  / | | |'  `.  |
    `--.  |    | |,'  | | |    |  |
   \___.' /    | /`---' / /    |  \__/
        osint | recon | security
      irc.hackint.org:6697/#sn0int
[+] Connecting to database
[+] Downloading public suffix list
[+] Downloading "GeoLite2-City.mmdb"
[+] Downloading "GeoLite2-ASN.mmdb"
[+] Loaded 0 modules
[*] No modules found, run quickstart to install default modules
[sn0int][default] >

Typing quickstart is going to get you a fair number of featured modules:

[sn0int][default] > quickstart
[+] Installing kpcyrd/asn
[+] Installing kpcyrd/ctlogs
[+] Installing kpcyrd/dns-resolve
[+] Installing kpcyrd/geoip
[+] Installing kpcyrd/hackertarget-subdomains
[+] Installing kpcyrd/otx-subdomains
[+] Installing kpcyrd/passive-spider
[+] Installing kpcyrd/pgp-keyserver
[+] Installing kpcyrd/threatminer-ipaddr
[+] Installing kpcyrd/threatminer-subdomains
[+] Installing kpcyrd/url-scan
[+] Installing kpcyrd/waybackurls
[+] Loaded 12 modules
[sn0int][default] >

  Adding something to scope

You probably want to separate your investigations so you should select a workspace where your results should go:

[sn0int][default] > workspace demo
[+] Connecting to database
[sn0int][demo] >

Next, we have to start somewhere and add the first entity to our scope:

[sn0int][demo] > add domain
Domain: example.com
[sn0int][demo] >

NOTE: There is a concept of a domain vs a subdomain. We are referring to a domain as everything that is a subdomain of a public suffix. For example, .com is a public suffix, which makes example.com a domain in sn0int terms. Every subdomain of that, like www.example.com, is referred to as a subdomain.

Note that example.com can be added as a subdomain as well since it can hold records. In that case, example.com is both the name of the dns zone, while also being an entity in that zone.

You can confirm this by running a select on the domains we now have:

[sn0int][demo] > select domains
#1, "example.com"
[sn0int][demo] >

Something we don't need right now, but is going to be useful later on is the ability to filter your entities:

[sn0int][demo] > select domains where id=1
#1, "example.com"
[sn0int][demo] >
[sn0int][demo] > select domains where value like %.com
#1, "example.com"
[sn0int][demo] >
[sn0int][demo] > select domains where ( value like e% and value like %m ) or false
#1, "example.com"
[sn0int][demo] >

NOTE: Almost all entities have a value column that holds the primary value of the entity.

  Running a module

Now that we have something to get started with, we can run our first module. First lets list all modules we have:

[sn0int][demo] > mod list
kpcyrd/asn (0.1.0)
    Run a asn lookup for an ip address
kpcyrd/ctlogs (0.1.0)
    Query certificate transparency logs to discover subdomains
kpcyrd/dns-resolve (0.1.0)
    Query subdomains to discovery ip addresses and verify the record is visible
kpcyrd/geoip (0.1.0)
    Run a geoip lookup for an ip address
kpcyrd/hackertarget-subdomains (0.1.0)
    Query hackertarget for subdomains of a domain
kpcyrd/otx-subdomains (0.1.0)
    Query alienvault otx passive dns for subdomains of a domain
kpcyrd/passive-spider (0.1.0)
    Scrape known http responses for urls
kpcyrd/pgp-keyserver (0.1.0)
    Query pgp keyserver for email addresses
kpcyrd/threatminer-ipaddr (0.1.0)
    Query ThreatMiner passive dns for subdomains of an ip address
kpcyrd/threatminer-subdomains (0.1.0)
    Query ThreatMiner passive dns for subdomains of a domain
kpcyrd/url-scan (0.1.0)
    Scan subdomains for websites
kpcyrd/waybackurls (0.1.0)
    Discover subdomains from wayback machine
[sn0int][demo] >

Let's start by querying certificate transparency logs:

[sn0int][demo] > use ctlogs
[sn0int][demo][kpcyrd/ctlogs] > run
[*] "example.com"                                     : Subdomain: "www.example.com"
[*] "example.com"                                     : Subdomain: "m.example.com"
[*] "example.com"                                     : Subdomain: "dev.example.com"
[*] "example.com"                                     : Subdomain: "products.example.com"
[*] "example.com"                                     : Subdomain: "support.example.com"
[+] Finished kpcyrd/ctlogs
[sn0int][demo][kpcyrd/ctlogs] >

Looks like we've discovered some subdomains here. It might be tempting to throw some of them in a browser but hold on, there's a more efficient way to approach this.

HINT: You can run the modules concurrently with run -j 8.

  Running followup modules on the results

A lot of time has been spent on the database part. While it sort of feels like a no-sql database we are actually enforcing a schema for a reason instead of just using generic dictionaries and calling it a day.

It's crucial that entities created by one module can be picked up by another module, like LEGOs. Let's continue with a module to query the dns records:

[sn0int][demo][kpcyrd/ctlogs] > use dns-resolve
[sn0int][demo][kpcyrd/dns-resolve] > run
[*] "www.example.com"                                 : Updating "www.example.com" (resolvable => true)
[*] "www.example.com"                                 : IpAddr: 93.184.216.34
[*] "www.example.com"                                 : "www.example.com" -> 93.184.216.34
[*] "m.example.com"                                   : Updating "m.example.com" (resolvable => false)
[*] "dev.example.com"                                 : Updating "dev.example.com" (resolvable => false)
[*] "products.example.com"                            : Updating "products.example.com" (resolvable => false)
[*] "support.example.com"                             : Updating "support.example.com" (resolvable => false)
[+] Finished kpcyrd/dns-resolve
[sn0int][demo][kpcyrd/dns-resolve] >

Two things happened here: We've discovered some IP addresses and added them to scope, and we also updated our subdomain entities with new information, since we now know which of them are resolvable and which aren't.

Let's run the next module, which is actually going to check for websites on them, but let's only target subdomains that we know are resolvable:

[sn0int][demo][kpcyrd/dns-resolve] > use url-scan
[sn0int][demo][kpcyrd/url-scan] > target
#1, "www.example.com"
    93.184.216.34
#2, "m.example.com"
#3, "dev.example.com"
#4, "products.example.com"
#5, "support.example.com"
[sn0int][demo][kpcyrd/url-scan] > target where resolvable
[+] 1 entities selected
[sn0int][demo][kpcyrd/url-scan] > target
#1, "www.example.com"
    93.184.216.34
[sn0int][demo][kpcyrd/url-scan] >

We can both preview and limit the targets that are going to be passed to the module with the target command. Once we are satisfied with our selection we can run this module:

[sn0int][demo][kpcyrd/url-scan] > run
[*] "www.example.com"                                 : Url: "http://www.example.com/" (200)
[*] "www.example.com"                                 : Url: "https://www.example.com/" (200)
[+] Finished kpcyrd/url-scan
[sn0int][demo][kpcyrd/url-scan] >

We've now probed both port 80 and port 443 for each subdomain and found two http responses this way. If you want a list of urls you may want to visit in your browser can now query them:

[sn0int][demo][kpcyrd/url-scan] > select urls
#1, "http://www.example.com/" (200)
#2, "https://www.example.com/" (200)
[sn0int][demo][kpcyrd/url-scan] >

  Unscoping entities

Something you are going to run into is that modules are too greedy and add things to the scope we are not interested in. You can delete them using the delete command, but those are likely picked up by a module again.

What you can do instead is setting a flag on an entity that removes it from our scope. This is done using the noscope command:

[sn0int][demo] > use ctlogs
[sn0int][demo][kpcyrd/ctlogs] > target
#1, "example.com"
[sn0int][demo][kpcyrd/ctlogs] > add domain
Domain: google.com
[sn0int][demo][kpcyrd/ctlogs] > target
#1, "example.com"
#2, "google.com"
[sn0int][demo][kpcyrd/ctlogs] > noscope domains where value=google.com
[+] Updated 1 rows
[sn0int][demo][kpcyrd/ctlogs] > target
#1, "example.com"
[sn0int][demo][kpcyrd/ctlogs] >

Entities that are unscoped are automatically ignored by all modules.

You can reverse this using the scope command:

[sn0int][demo][kpcyrd/ctlogs] > target
#1, "example.com"
[sn0int][demo][kpcyrd/ctlogs] > scope domains where true
[+] Updated 2 rows
[sn0int][demo][kpcyrd/ctlogs] > target
#1, "example.com"
#2, "google.com"
[sn0int][demo][kpcyrd/ctlogs] >

HINT: All entities have this field, you can refer to it in queries using unscoped=1.

FUNCTION REFERENCE

  clear_err

Clear the last recorded error from the internal state. See also last_err.

if last_err() then
    -- ignore this error
    clear_err()
end

  db_add

Add an entity to the database or update it if it already exists. This function may fail or return nil. See db_add for details.

domain_id = db_add('domain', {
    value='example.com',
})

  db_select

Checks if a target is in scope. If non-nil is returned, this entity is in scope. This function may fail. See db_select for details.

domain_id = db_select('domain', 'example.com')
if domain_id ~= nil then
    -- do something
end

  db_update

Update an entity in the database. This function may fail. See db_update for details.

db_update('ipaddr', arg, {
    asn=lookup['asn'],
    as_org=lookup['as_org'],
})

  dns

Resolve a dns record. In the dns response contains an error, x['error'] is set and x['success'] is nil. Otherwise, x['success'] contains a list of records. This function may fail.

x = dns('example.com', 'A')
if last_err() then return end

  error

Log an error to the terminal.

error('ohai')

  asn_lookup

Run an ASN lookup for a given ip address. The function returns asn and as_org. This function may fail.

lookup = asn_lookup('1.1.1.1')
if last_err() then return end

  geoip_lookup

Run a geoip lookup for a given ip address. The function returns:

•   continent

•   continent_code

•   country

•   country_code

•   city

•   latitude

•   longitude

    This function may fail.

    lookup = geoip_lookup('1.1.1.1')
    if last_err() then return end

  html_select

Parses an html document and returns the first element that matches the css selector. The return value is a table with text being the inner text and attrs being a table of the elements attributes.

csrf = html_select(html, 'input[name="csrf"]')
token = csrf["attrs"]["value"]

  html_select_list

Same as html_select but returns all matches instead of the first one.

html_select_list(html, 'input[name="csrf"]')

  http_mksession

Create a session object. This is similar to requests.Session in python-requests and keeps track of cookies.

session = http_mksession()

  http_request

Prepares an http request. The first argument is the session reference and cookies from that session are copied into the request. After the request has been sent, the cookies from the response are copied back into the session.

The next arguments are the method, the url and additional options. Please note that you still need to specify an empty table {} even if no options are set. The following options are available:

query
    A map of query parameters that should be set on the url

headers
    A map of headers that should be set

basic_auth
    Configure the basic auth header with {"user, "password"}

user_agent
    Overwrite the default user agent with a string

json
    The request body that should be json encoded

form
    The request body that should be form encoded

body
    The raw request body as string

    This function may fail.

    req = http_request(session, 'POST', 'https://httpbin.org/post', {
        json={
            user=user,
            password=password,
        }
    })
    resp = http_send(req)
    if last_err() then return end
    if resp["status"] ~= 200 then return "invalid status code" end

  http_send

Send the request that has been built with http_request. Returns a table with the following keys:

status
    The http status code

headers
    A table of headers

text
    The response body as string

    req = http_request(session, 'POST', 'https://httpbin.org/post', {
        json={
            user=user,
            password=password,
        }
    })
    resp = http_send(req)
    if last_err() then return end
    if resp["status"] ~= 200 then return "invalid status code" end

  info

Log an info to the terminal.

info('ohai')

  json_decode

Decode a lua value from a json string.

json_decode("{\"data\":{\"password\":\"fizz\",\"user\":\"bar\"},\"list\":[1,3,3,7]}")

  json_decode_stream

Very similar to json_decode, but works with multiple json objects directly concatenated to each other or separated by newlines.

json_decode_stream("{\"data\":1}{\"data\":2}")

  json_encode

Encode a datastructure into a string.

x = json_encode({
    some=1,
    fancy={
        data='structures',
    }
})
print(x)

  last_err

Returns infos about the last error we've observed, if any. Returns nil otherwise.

if last_err() then
    -- Something went wrong, abort
    return
end

  pgp_pubkey

Same as pgp_pubkey_armored, but without the unarmor step.

  pgp_pubkey_armored

Extract uids out of a rfc 4880 pgp public key. This function may fail.

key = pgp_pubkey_armored([===[
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2
mQENBFu6q90BCADgD7Q9aH5683yt7hzPktDkAUNAZJHwYhUNeyGK43frPyDRWQmq
N+oXTfiYWLQN+d7KNBTnF9uwyBdaLM7SH44lLNYo8W09mVM2eK+wt19uf5HYNgAE
8la45QLo/ce9CQVe1a4oXNWq6l0FOY7M+wLe+G2wMwz8RXGgwd/qQp4/PB5YpUhx
nAnzClxvwymrL6BQXsRcKSMSD5bIzIv95n105CvW5Hql7JR9zgOR+gHqVOH8HBUc
ZxMumrTM6aKLgAhgM8Sn36gCFOfjlG1b1OFLZhUtgro/nnEOmAurRsCZy8M5h8QM
FpZChIH8kgHs90F/CCvGjMq3qvWcH8ZsPUizABEBAAG0NUhhbnMgQWNrZXIgKGV4
YW1wbGUgY29tbWVudCkgPGhhbnMuYWNrZXJAZXhhbXBsZS5jb20+iQFOBBMBCAA4
FiEEyzeO1eEwbB03hcqBM00IodGdlj8FAlu6q90CGwMFCwkIBwIGFQgJCgsCBBYC
AwECHgECF4AACgkQM00IodGdlj/AJQgAjmk+iP5b7Jt7+f+lU4Oprlf3f3DG/uh5
Ge6MjV7cvtxlhZJRD5hxGt9RwwnEp61TBSbrem288pM89ilQfTNe0wUr9OzwWzh/
8Ngl5iWnD2ah3Mpi5R1V/YMNf2cnwVjqNvfkRHdNc43pZOkC2GoiTUn0QY0UBpOW
ZMN3//ANi6ZtiK/L0IZQND/gKvOzu/4tfaJeBl26T3cVYj53p3G3jhlb92vVa8SR
uL3S3bzd1h5snDgU1uXHmNHGbhkEc4KUneQ0V9/bdZrg6OzFAfM1ghgfoId+YpQH
er9L26ISL3QF58wdEXfIdHYEmMlANjBMO2cUlQXgONuCgkMuY7GBmrkBDQRbuqvd
AQgA41jqCumCxYV0NdSYNnTSSDRyd69dOUYCAPT80iZ739s7KKJS9X9KVfGmDjfi
u2RcfR/KYj53HoyOm4Pm/+ONN8De4ktzXpIpJxGC+O8NBvd9vkboAS6qnCjK7KVE
r91ymxxVKp2dzZvVfpIjWVZR5i2EAvS5vw8UK4gL8ALH+S9leJFZrQWcgyoJOJzH
Rzr9pesX2HvdgcNG1O6QUArlsnsTnqpi/hu7tQa8tifBpWDeArOA23Y2DgeehdDF
lSU/8KD4J+AkFrWWlcTaMsvSChXQkCHEMRIcSOfXtdpX5KJSE7UBQdD1opm+mR79
VeHnuJAAVZZtUZmJA7pjdKykYQARAQABiQE2BBgBCAAgFiEEyzeO1eEwbB03hcqB
M00IodGdlj8FAlu6q90CGwwACgkQM00IodGdlj8bMAf+Lq3Qive4vcrCTT4IgvVj
arOACdcbtt5RhVBTimT19rDWNH+m+PfPjo3FSlBj5cm70KAXUS2LBFFxhakTZ/Mq
cQroWZpVbBxj4kipEVVJZFdUZQaDERJql0xYGOQrNMQ4JGqJ84BRrtOExjSqo41K
hAhNe+bwPGH9/Igiixc4tH07xa7TOy4MyJv/6gpbHy/lW1hqpCAgM5fT/im5/6QF
k0tED6vIuc54IWiOmwCnjZiQnJ8uCwEu+cuJ5Exwy9CNERLp5v0y4eG+0E+at9j/
macOg39qf09t53pTqe9dWv5NIi319TeBsKZ2lb0crrQjsbHqk0DAUwgQuoANqLku
vA==
=kRIv
-----END PGP PUBLIC KEY BLOCK-----
]===])
if last_err() then return end
print(key)

  print

Write something directly to the terminal.

print({
    some=1,
    fancy={
        data='structures',
    }
})

WARNING: This function writes directly to the terminal and can interfere with other terminal features. This function should be used during development only.

  psl_domain_from_dns_name

Returns the parent domain according to the public suffix list. For www.a.b.c.d.example.com this is going to be example.com.

domain = psl_domain_from_dns_name('www.a.b.c.d.example.com')
print(domain == 'example.com')

  regex_find

Apply a regex to some text. Returns nil if the regex didn't match and the capture groups if it did.

m = regex_find(".(.)", "abcdef")
if m == nil then
    print('No captures')
end
print(m[1] == 'ab')
print(m[2] == 'b')

  regex_find_all

Same as regex_find, but returns all matches.

m = regex_find_all(".(.)", "abcdef")
print(m[1][1] == 'ab')
print(m[1][2] == 'b')
print(m[2][1] == 'cd')
print(m[2][2] == 'd')
print(m[3][1] == 'ef')
print(m[3][2] == 'f')

  sleep

Pause the current program for the specified number of seconds. This is usually only used for debugging.

sleep(1)

  status

Update the label of the progress indicator.

status('ohai')

  url_join

Join a relative link to an absolute link. If both links are absolute we just return the first one:

x = url_join('https://example.com/x', '/foo')
print(x == 'https://example.com/foo')
x = url_join('https://example.com/x', 'https://github.com/')
print(x == 'https://github.com/')

  url_parse

Parse a url into its components. The following components are returned:

•   scheme

•   host

•   port

•   path

•   query

•   fragment

•   params

    url = url_parse("https://example.com")
    print(url['scheme'] == 'https')
    print(url['host'] == 'example.com')
    print(url['path'] == '/')

  utf8_decode

Decodes a list of bytes/numbers into a string. This function might fail.

x = utf8_decode({65, 65, 65, 65})
if last_err() then return end
print(x == 'AAAA')

  x509_parse_pem

Parse a pem encoded certificate. This function might fail.

x = x509_parse_pem([[-----BEGIN CERTIFICATE-----
MIID9DCCA3qgAwIBAgIQBWzetBRl/ycHFsBukRYuGTAKBggqhkjOPQQDAjBMMQsw
CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSYwJAYDVQQDEx1EaWdp
Q2VydCBFQ0MgU2VjdXJlIFNlcnZlciBDQTAeFw0xODAzMzAwMDAwMDBaFw0yMDAz
MjUxMjAwMDBaMGwxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMN
U2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEdMBsGA1UE
AwwUKi5jbG91ZGZsYXJlLWRucy5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AASyRQsxrFBjziHmfDQjGsXBU0WWl3oxh7vg6h2V9f8lBMp18PY/td9R6VvJPa20
AwVzIJI+dL6OSxviaIZEbmK7o4ICHDCCAhgwHwYDVR0jBBgwFoAUo53mH/naOU/A
buiRy5Wl2jHiCp8wHQYDVR0OBBYEFN+XTeVDs7BBp0LykM+Jf64SV4ThMGMGA1Ud
EQRcMFqCFCouY2xvdWRmbGFyZS1kbnMuY29thwQBAQEBhwQBAAABghJjbG91ZGZs
YXJlLWRucy5jb22HECYGRwBHAAAAAAAAAAAAERGHECYGRwBHAAAAAAAAAAAAEAEw
DgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBp
BgNVHR8EYjBgMC6gLKAqhihodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vc3NjYS1l
Y2MtZzEuY3JsMC6gLKAqhihodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vc3NjYS1l
Y2MtZzEuY3JsMEwGA1UdIARFMEMwNwYJYIZIAYb9bAEBMCowKAYIKwYBBQUHAgEW
HGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwCAYGZ4EMAQICMHsGCCsGAQUF
BwEBBG8wbTAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQuY29tMEUG
CCsGAQUFBzAChjlodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGlnaUNlcnRF
Q0NTZWN1cmVTZXJ2ZXJDQS5jcnQwDAYDVR0TAQH/BAIwADAKBggqhkjOPQQDAgNo
ADBlAjEAjoyy2Ogh1i1/Kh9+psMc1OChlQIvQF6AkojZS8yliar6m8q5nqC3qe0h
HR0fExwLAjAueWRnHX4QJ9loqMhsPk3NB0Cs0mStsNDNG6/DpCYw7XmjoG3y1LS7
ZkZZmqNn2Q8=
-----END CERTIFICATE-----
]])
if last_err() then return end
print(x)

AUTHOR
kpcyrd

COPYRIGHT
2018, kpcyrd