Skip to content
Shodan Book

InternetDB

The InternetDB dataset provides a single-file, drop-in Shodan SQLite database that contains information about recently-seen banners for devices and lets you do fast IP lookups. Similar to a local GeoIP database but instead of providing location information we provide network information. The following properties are currently provided by InternetDB:

  • Open ports
  • Vulnerabilities
  • Hostnames
  • CPEs
  • Tags

The information is significantly smaller than the full banner data but there are a few advantages to InternetDB:

  1. Very Fast
    The entire database is small enough to fit into memory on a single computer which means that even on commodity hardware you can easily do more than 60,000 IP lookups per second.
  2. No infrastructure required
    It's a single file that doesn't require a database service.
  3. Works Offline
    The database fits on a USB thumb drive and can be used without Internet access.

Performance

REST APIREST API (Batch IP)InternetDB (SQLite)
1 IP per second100 IPs per second60,000+ IPs per second

The Shodan API allows 1 request per second. In practice, it can take longer than 1 second to grab the data for a host depending on latency, number of banners and other factors. For the purpose of this comparison, we assumed that you can get the maximum of 1 IP per second. The InternetDB was loaded on a commodity SSD and we performed 10,000 IP lookups using individual queries in the form of:

SELECT * FROM data WHERE ip=?

It could be further optimized by loading the entire database into memory, putting it on faster drive or batching the IP lookups.

Quickstart

For most systems, there will not be much to install as SQLite is readily available across platforms. You only need a way to download the InternetDB SQLite file and a cronjob to periodically update it. Here’s how you could do it using the official Shodan CLI:

  1. Install the Shodan command-line interface (CLI):
    Terminal window
    pip install --user shodan
  2. Initialize the CLI using your Shodan API key. You can get your API key from the Shodan account website:
    Terminal window
    shodan init API_KEY
  3. Download the file:
    Terminal window
    shodan data download internetdb internetdb.sqlite.bz2
  4. Rename and uncompress the file:
    Terminal window
    mv internetdb-internetdb.sqlite.bz2 internetdb.sqlite.bz2 bunzip2 internetdb.sqlite.bz2

Alternatively, the following script can be run once a day to ensure the latest copy of InternetDB is locally available. And the script only requires curl to download the file - no additional dependencies needed:

https://gist.github.com/achillean/c1ab2e396adb632eb016fbe210c8b0b3

The SQLite file can then be queried using Python or any programming language that supports SQLite:

import sqlite3


# Load the SQLite database
con = sqlite3.connect('internetdb.sqlite')
cur = con.cursor()


# Grab information about 1.1.1.1
cur.execute("SELECT ip,ports,hostnames,tags,vulns,cpes FROM data WHERE ip=?", ("1.1.1.1",)
)
info = cur.fetchone()
print(info)

Integration: Zeek

Having the database locally available lets you do the following in real-time:

  • Enrich netflow
  • Block connections from IPs with known vulnerabilities or that have been compromised
  • Block connections from 3rd-party VPNs, proxies
  • Block outgoing connections to insecure IPs

Shodan provides an integration with Zeek to accomplish the above and we've provided a step-by-step guide on getting it setup.

For example, the following script drops connections coming from proxies:

event new_connection(c: connection) {
    local b = InternetDB::lookup_internetdb_sqlite(c$id$resp_h);
    if ( b?$tags ){
        if ( "proxy" in b$tags ){
            NetControl::drop_connection(c$id, 1min);
        }
    }
}