Lens BigQuery

Lens BigQuery is a public dataset that contains all Lens public data.

Overview

Lens Protocol utilizes indexers to snapshot and transform data into a PostgreSQL database, accessible via the Lens GraphQL API. This API is ideal for apps requiring real-time data, like displaying a feed or a profile upon user login. However, the challenge arises in bulk data retrieval, which is essential for machine learning training, analytics, or data profiling.

For these use cases, Lens has published its entire social graph to a public BigQuery dataset. This allows for bulk querying of data in any desired manner, making it accessible to anyone without the need to build their own complex infrastructure or indexers. It brings the entire Lens graph into the cloud for convenient querying.

Typically, the data is delayed by about 15 minutes. This delay is due to the synchronization process, which updates at the fastest possible interval of 15 minutes.

Use Cases

With access to this dataset, you can:

Conduct analytics, such as creating analytics dashboards.
Extract large data volumes for machine learning applications, like discovering trends in publications.
Develop feeds with curated content.

This enables custom feature development on Lens, beyond what the API offers. It's an invitation to think creatively with the available data and build something remarkable.

Tables

To understand how to query these public datasets from BigQuery, you can refer to these guides. Queries can be executed via the Google Cloud Console or through BigQuery client libraries, which support numerous programming languages. The simplest starting point is using the Analytic Hub, where you can run queries directly.

Datasets

For querying data from the Lens Protocol, use these fully-qualified table names and replace INSERT_YOUR_TABLE_NAME_HERE with the specific table name you're interested in:

Mainnet: lens-public-data.v2_polygon.INSERT_YOUR_TABLE_NAME_HERE
Testnet: lens-public-data.v2_amoy.INSERT_YOUR_TABLE_NAME_HERE

Examples

To access all publications on Polygon:

SELECT * FROM lens-public-data.v2_polygon.publication_record LIMIT 10

To list all available tables on Polygon:

SELECT table_name FROM lens-public-data.v2_polygon.INFORMATION_SCHEMA.TABLES

Please note you MUST query this from the US region. If you try to use the EU region it will not be able to find it.