Metadata and contents are extracted using ipfs-tika, searching is done using ElasticSearch 5, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.
A preliminary start at providing a minimal amount of documentation can be found in the docs folder.
Please find us on our Freenode/Riot/Matrix channel #ipfssearch.
ipfs-search provides the daily snapshot for all of the indexed data using
To learn more about downloading and restoring snapshots, read docs
Building a search engine like this takes a considerable amount of resources (money and TLC).
If you are able to help out with either of them, mail us at email@example.com or find us at #ipfssearch on Freenode (or #ipfs-search:chat.weho.st on Matrix).
The following configuration options can be overridden by environment variables:
or by using environment variables.
$ go get ./...
The most convenient way to run the crawler is through Docker. Simply run:
This will start the crawler and all its dependencies but will not (yet) launch the sniffer or search API. Hashes can be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example: