Skip to content
Shodan Book

DNSDB

The DNSDB dataset contains 3 types of files:

  • Weekly SQLite database (dnsdb.sqlite.zst)
    The SQLite file contains 2 tables to lookup DNS records based on hostname or lookup hostnames by IP address. The data is based on the most recent 30 days of data.
  • Daily CSV (yyyy-mm-dd.csv.zst)
    The daily CSV contains all the DNS records collected for the given date.
  • Weekly CSV (dnsdb.csv.gz)
    The CSV contains the DNS records collected the past 30 days.

If you’re looking for a quick way to get started with DNSDB then we would recommend the SQLite file. It comes pre-indexed for fast lookups by hostname or IP address and can be synchronized weekly to stay up to date. The schema for the SQLite file is:

CREATE TABLE IF NOT EXISTS hostnames (
    hostname TEXT,
    domain TEXT,
    type TEXT,
    value TEXT
);
CREATE TABLE IF NOT EXISTS ip_hostname (
    ip TEXT,
    hostname TEXT
);
CREATE INDEX IF NOT EXISTS domain_index ON hostnames (domain);
CREATE INDEX IF NOT EXISTS ip_index ON ip_hostname (ip);

If you want to receive more frequent updates than weekly and are comfortable managing a local database server then we would recommend the daily DNS files.

The below examples use the SQLite file to perform queries without requiring a database server or external API:

Examples

Get a list of subdomains

The SQL query to fetch the information is simple:

SELECT hostname FROM hostnames WHERE domain='shodan.io';

To give you an idea of the performance of the dnsdb.sqlite file here are some numbers for grabbing all the data for the amazonaws.com domain:

Terminal window
$ time sqlite3 dnsdb.sqlite "select count(*) from hostnames where domain='amazonaws.com'"
4267421
real    0m0.193s
user    0m0.142s
sys 0m0.051s


$ time sqlite3 dnsdb.sqlite "select * from hostnames where domain='amazonaws.com'" > /dev/null
real    0m1.692s
user    0m1.432s
sys 0m0.260s

There are around 4.2 million records for the domain and it takes around 1.7 seconds to iterate over all of them using a mid-range laptop.

Finding other websites hosted on the same server

DNSDB can be used to quickly identify other websites that are using the same public IP. For example, lets see which IPs that shodan.io is using:

sqlite> select * from hostnames where hostname='www' and domain='shodan.io';
www|shodan.io|A|104.18.12.238
www|shodan.io|A|104.18.13.238
www|shodan.io|AAAA|2606:4700::6812:cee
www|shodan.io|AAAA|2606:4700::6812:dee

And now we can query the ip_hostnames table to quickly find all the other hostnames associated with those IPs:

sqlite> select * from ip_hostname where ip='104.18.12.238';
104.18.12.238|alert.co.za
104.18.12.238|dev-api.alert.co.za
104.18.12.238|east-rand.staging.alert.co.za
104.18.12.238|johannesburg.staging.alert.co.za
104.18.12.238|staging.alert.co.za
104.18.12.238|www.alert.co.za
104.18.12.238|advancedwindowsystemsllc.com
104.18.12.238|adbox004.com
104.18.12.238|exitwidget-id.com
104.18.12.238|www.exitwidget-id.com
104.18.12.238|api.ext.fourthline.com
104.18.12.238|mapi.ext.fourthline.com
...

The results above have been truncated because Shodan uses Cloudflare as its CDN so there are a lot of other websites with the same public IP.

Data Refresh Rate

The list of hostnames and domains that DNSDB fetches information for is based on a variety of algorithms that follow common OSINT best practices. Those algorithms run continuously and are constantly adding new hostnames/ domains to the list of targets.

We refresh the DNS records for those targets in the following timeframes:

  • Monitored domains/ hostnames have their A and AAAA records refreshed every 3 hours
  • Weekly refresh of all record types
  • Semi-weekly refresh of Aand AAAArecords
  • Daily refresh for A and AAAAfor hostnames on priority list
    ⇒ To add hostnames/ domains to the priority list please send list of hostnames enterprise@shodan.io with subject “DNSDB: Priority List”