Monitor usage¶

Bigdata.com API subscriptions have two quotas:

API Query Unit: Allow customers to retrieve data from Bigdata.com.
Pages of uploaded files: Allow customers to upload files to Bigdata.com.
- PDF format files
- Other format files (CSV, XML, JSON, HTML, TXT, DOCX)
  
  In standard contracts the platform considers a page as group of 3000 characters.

Note

We encourage you to monitor usage and contact us if you would like assistance in choosing the right subscription plan for your organization.

You can monitor usage at the Subscription level or at a Search level.

Tip

We recommend to try this how-to guide on Google Colab. Otherwise follow Prerequisites instructions to set up your Python SDK environment in your local environment.

Subscription level¶

Use the method get_details to retrieve subscription details:

from bigdata_client import Bigdata

bigdata = Bigdata()
subscription_details = bigdata.subscription.get_details()

API Query Unit¶

Bigdata.com measures the amount of retrieved data with API Query Units; each unit allows retrieval of 10 text chunks.

subscription_details = bigdata.subscription.get_details()
print(f"API Query Unit monitoring")
print(f"Total: {subscription_details.organization_quota.query_unit.total}")
print(f"Used: {subscription_details.organization_quota.query_unit.used}")
print(f"Remaining: {subscription_details.organization_quota.query_unit.remaining}")

Output:

API Query Unit monitoring
Total: 50000
Used: 2025
Remaining: 47975

How many API Query Units consume a Search?

The method search.run() accepts a parameter to specify the number of documents or chunks to retrieve. Every ten retrieved chunks count as one API Query Unit.

How can I control the API Query Units usage per Search?

You can control the usage by specifying the number of chunks to retrieve with the parameter ChunkLimit:

search.run(ChunkLimit(100)) will retrieve a maximum of 100 chunks and therefore will consume a maximum of 10 API Query Units. The response might contain a smaller number of chunks due to discarding duplicates, so the usage could be lower.

Check the how-to guide Retrieve limited chunks for more details.

How can I see the API Query Unit usage per Search?

Check the usage of each search run at the Search level.

Pages of uploaded files¶

Bigdata.com measures the amount of uploaded files in pages.

PDF format¶

subscription_details = bigdata.subscription.get_details()
print(f"PDF files pages monitoring")
print(f"Total: {subscription_details.organization_quota.pdf_upload_pages.total}")
print(f"Used: {subscription_details.organization_quota.pdf_upload_pages.used}")
print(f"Remaining: {subscription_details.organization_quota.pdf_upload_pages.remaining}")

Output:

PDF files pages monitoring
Total: 20000
Used: 13923
Remaining: 6077

Other format¶

subscription_details = bigdata.subscription.get_details()
# Generic file format (CSV, XML, JSON, HTML, TXT, DOCX…)
print(f"Generic file pages monitoring")
print(f"Total: {subscription_details.organization_quota.file_upload_pages.total}")
print(f"Used: {subscription_details.organization_quota.file_upload_pages.used}")
print(f"Remaining: {subscription_details.organization_quota.file_upload_pages.remaining}")

Output:

Generic file pages monitoring
Total: 20000
Used: 13923
Remaining: 6077

Search level¶

Each new search tracks the amount of retrieved data, and you can consult it at any time with the method get_usage()

Initially, the usage of a new search is 0

from bigdata_client.query import Keyword

some_search = bigdata.search.new(query=Keyword("AI in finance"))
print(f"Initial search usage {some_search.get_usage()}")

After executing the method run(), the method get_usage() returns the used Query Units.

The response might contain a smaller number of chunks due to discarding duplicates, so the usage could be lower. Check the how-to guide Retrieve limited chunks for more details.

from bigdata_client.search import ChunkLimit

some_search.run(ChunkLimit(100))
print(f"Current search usage {some_search.get_usage()}")

Please get in touch with your Account Manager or support@bigdata.com if you have any questions or would like guidance in selecting the best subscription for your needs.