Handling items of different types

A data source could contain items of different types, which can be handled with some preparation:

  • Create one schema per item type choosing homogeneous items basic_json_schema('job_key', items_number=[0,1)

  • Pass a filter argument to choose items of one type - Arche(source='000000/000/0', schema=schema, filter=[("_type", "=", ["ItemType"])])

  • Repeat the previous steps for each item type

The library also supports other API arguments, such as count, start. The complete list of arguments is available in Scrapinghub Python API documentation:

  • https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html#scrapinghub.client.items.Items.iter

  • https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html#scrapinghub.client.collections.Collection.iter