Installation
The first thing you need to do is to download our python package from PyPI, you can install it with pip
Copy pip install picsellia Dependencies
Here is the several packages that will be installed along the picsellia package, they are used to perform mathematical operations, array or image manipulations and HTTP requests
Client
To connect your code to Picsell.ia, you must initialize our client.
Copy from picsellia.client import Client
api_token = '4d388e237d10b8a19a93517ffbe7ea32ee7f4787'
clt = Client(api_token) You will be greeted by this nice message (Pierre-Nicolas is my name, you should see your username here, unless you are named Pierre-Nicolas too 👀 ) :
Copy Hi Pierre-Nicolas, welcome back. Datalake
You need to instantiate a Datalake object in order to interact with your pictures or datasets.
__init__
If you need to interact with your datalake, you must initialize the Datalake class.
It is a subclass of the Client.
Copy from picsellia.client import Client
datalake = Client.Datalake(
api_token=None,
organization=None
) Arguments:
api_token (string) Your personal API token (Find it in your profile page 🔥)
organization (string, optional) the name of the organization you want to work with (None default to your organization)
Returns:
None
upload
This upload method allows you to upload your Dataset, (images and annotations or annotations only).
Copy datalake.upload(
name: str=None,
version: str=None,
imgdir: str=None, -> "path/to/imgdir"
ann_path: str=None, -> "path/to/annotations"
ann_format: str=None, -> "COCO", "PASCAL-VOC", "PICSELLIA"
tags: List[str]=[],
nb_jobs: int=5,
rectangle: bool=False): Arguments
name (string, required) Name of the dataset to be created or fetched if you want to upload only annotations
version (string, optionnal) Version of the dataset to be fetched, if None we will fetch the latest created with this name.
imgdir (string, optionnal) Path to the directory of the pictures to upload, leave blank if you just want to upload the annotations.
ann_path (string, required) Path to the annotations files, it can be a directory or a file.
ann_format (string, required) you can upload COCO format, PASCAL-VOC or PICSELLIA
tags (list, optionnal) List of tags to add to the uploaded images, if None tags will be the upload date.
nb_jobs (int, optionnal) Number of process to use to upload, put nb_jobs=-1 if you want to use all available processes.
rectangle (bool, optionnal) set to True if you want to force to upload bbox annotations when both polygons and bbox are available in your annotation files.
Picture
The picture object allow you to interact with your assets only.
__init__
If you need to interact with your experiments, you must initialize the Picture class.
It is a subclass of the Datalake.
Copy from picsellia.client import Client
picture = Client.Datalake.Picture(
api_token=None,
organization=None
) Arguments:
api_token (string) Your personal API token (Find it in your profile page 🔥)
organization (string, optional) the name of the organization you want to work with (None default to your organization)
upload
To upload assets to your lake
Copy picture.upload(filepath=None, tags=[], source='sdk') Arguments:
filtepath (string or list) Either one filepath pointing to the asset to upload or a list of path.
tags (list, optional) the list of tags to attach to the upload assets
source (string, optional) Specify the source of the upload, default is "sdk"
list
Returns:
A list containing the pictures objects for your datalake.
Copy {'pictures': [{'picture_id': '25d76bee-a6d3-43d7-8620-6ff18f7a5557',
'internal_key': '15288614-bedb-4cab-97c1-23684cf9c761.jpg',
'external_url': 'GE_121.jpg',
'creation_date': '2021-02-07',
'height': 310,
'width': 322,
'tag': []},
{'picture_id': 'd1cf0d96-5c05-4fb4-aa4a-5e90f3c748da',
'internal_key': 'ed6e12e3-0db3-461f-bcd9-54d48509680b.jpg',
'external_url': 'GE_55.jpg',
'creation_date': '2021-02-07',
'height': 2908,
'width': 4800,
'tag': []},
{'picture_id': '716e45a8-09f6-4ec5-9dd1-29c313ae2cdf',
'internal_key': 'a9597e66-584d-4568-b1cc-31b951154edd.jpg',
'external_url': 'GE_309.jpg',
'creation_date': '2021-02-07',
'height': 208,
'width': 254,
'tag': []},
{'picture_id': '88d2b82d-2a38-4c30-912b-79474a617072',
'internal_key': 'c83493f7-ce61-4a0b-8166-54698d071792.jpg',
'external_url': 'Test85.jpg',
'creation_date': '2021-02-07',
'height': 663,
'width': 710,
'tag': []},
{'picture_id': 'd9d4684f-a2d1-4431-93e7-dce352aff471',
'internal_key': 'b7a8bb6b-d3f2-46d5-9599-dfb9b4d2f1cd.jpg',
'external_url': 'GE_466.jpg',
'creation_date': '2021-02-07',
'height': 462,
'width': 520,
'tag': []},
} fetch
Fetch images with corresponding tags
Copy pictures.fetch(
quantity=1,
tags=["drone", "coco"],
) Parameters:
quantity (float, optional) the percentage of assets to fetch ( 1 meaning 100%)
tags (list, required) a list of tags used to search in your Datalake
Returns:
The list of all the fetched assets
Copy [{'picture_id': '8b536f4c-c95b-4f5f-afbe-a9f31242a235',
'internal_key': '51ee5ee9-5176-4e98-b173-0687ed6c7b2f.jpg',
'external_url': '9999966_00000_d_0000055.jpg',
'creation_date': '2021-02-07',
'height': 1050,
'width': 1400,
'tag': ['drone', 'coco', 'vizdrone']},
{'picture_id': '426ce7bd-7535-4fe5-80cd-c41e07f84c99',
'internal_key': '7f4f1b60-d1bb-4458-b3bb-5f3d01a8f7eb.jpg',
'external_url': '9999955_00000_d_0000312.jpg',
'creation_date': '2021-02-07',
'height': 788,
'width': 1400,
'tag': ['drone', 'coco', 'vizdrone']},
{'picture_id': '320e69fc-964a-478e-b689-05351213578e',
'internal_key': '5aa4036e-8050-4fef-9c3c-af9ba46db511.jpg',
'external_url': '9999955_00000_d_0000043.jpg',
'creation_date': '2021-02-07',
'height': 788,
'width': 1400,
'tag': ['drone', 'coco', 'vizdrone']},
{'picture_id': 'bed1ddab-7cf1-460b-99a8-c4125612caa3',
'internal_key': 'f73fedbf-d87e-4483-859a-77a3f8e38702.jpg',
'external_url': '9999982_00000_d_0000167.jpg',
'creation_date': '2021-02-07',
'height': 1050,
'width': 1400,
'tag': ['drone', 'coco', 'vizdrone']},
{'picture_id': '0bc695d1-03bb-48ea-bd89-5bef5bf02c23',
'internal_key': '98ac39c5-a157-4a5e-bafc-a8399b90f230.jpg',
'external_url': '9999974_00000_d_0000049.jpg',
'creation_date': '2021-02-07',
'height': 1078,
'width': 1916,
'tag': ['drone', 'coco', 'vizdrone']},
] status
Once, you have fetched pictures, you can call status method to visualize the number of assets fetched
Return:
Copy Number of Assets selected : 1472 delete
Delete the list of pictures
Copy pictures.delete(
pictures=None
) Arguments:
pictures (list, optional) The list of pictures to delete from your lake, if none will delete the latest fetched pictures.
Returns:
None
Add tags to selected pictures
Copy pictures.add_tags(
pictures=[],
tags=["tag_to_add"]
) Arguments:
pictures (list, optional) The list of pictures to selected from your lake, if none will add tags to the last fetched pictures.
tags (list, required) The list of tags to add to the selected pictures
Returns:
None
Remove tags from selected pictures
Copy pictures.remove_tags(
pictures=[],
tags=["tag_to_add"]
) Arguments:
pictures (list, optional) The list of pictures to selected from your lake, if none will remove tags from the last fetched pictures.
tags (list, required) The list of tags to delete from the selected pictures
Returns:
None
Dataset
The dataset object allow you to interact with your dataset ( annotations, labels, question and answers ).
__init__
If you need to interact with your experiments, you must initialize the Dataset class.
It is a subclass of the Datalake.
Copy from picsellia.client import Client
dataset = Client.Datalake.Dataset(
api_token=None,
organization=None
) Arguments:
api_token (string) Your personal API token (Find it in your profile page 🔥)
organization (string, optional) the name of the organization you want to work with (None default to your organization)
list
Returns:
A list containing the dataset objects for your account.
Copy -------------
Dataset Name: GoogleEarthShip
Dataset Version: first
Nb Assets: 793
-------------
-------------
Dataset Name: VizDrone2017
Dataset Version: first
Nb Assets: 6470
-------------
-------------
Dataset Name: FaceMaskDetection
Dataset Version: first
Nb Assets: 6024
-------------
-------------
Dataset Name: TrashDataset
Dataset Version: first
Nb Assets: 1435
------------- fetch
Fetch dataset with its name and version
Copy dataset.fetch(
name="myDataset",
version="latest",
) Parameters:
name (string, optional) the name of the dataset to fetch
version (string, optional) the version of the dataset to fetch, if None, the client will fetch latest
Returns:
A Dataset object relative to the fetched dataset
Copy dataset = dataset.fetch(name="myDataset",version="latest")
print(dataset)
{
"dataset_id": "9061846f-597a-47d6-9711-7f75671841a2",
"dataset_name": "myDataset",
"version": "latest",
"size": 128,
"description": "None",
"private": true
} create
Copy dataset.create(
name="myDataset",
description='',
private=True,
pictures=[]
)
Parameters:
name (string, optional) the name ot the dataset to create
description (string, optional) the description of the dataset to create
private (bool, optional) If True, your dataset will be accessible to anyone
pictures (list, required) The list of pictures to attach to your dataset
Return:
the id of the created dataset
new_version
Copy dataset.new_version(
name="myDataset",
version='newVersion',
from_version='latest',
pictures=[]
)
Parameters:
name (string, optional) the name of the dataset
version (string, optional) the version name of the dataset to create
from_version (string, optional)The origin version for your new version, if None we'll create a new version from the latest version
pictures (list, required) The list of pictures to attach to your version
Return:
None
create_labels
Sets up the labels (tools for drawing bounding-boxes, polygons...) for your dataset.
Copy dataset.create_labels(
name=None,
ann_type=None
) Arguments:
name (str, required) name of the label want to set up (e.g. car, bird, plane...)
ann_type (str, required) type of shape that will be used for annotations :
'rectangle': bounding-boxes for object-detection
'polygon': polygons for segmentation
Returns:
None
list_pictures
Get the list of all the images in the fetched dataset
Copy dataset.list_pictures(
dataset_id=None
) Arguments:
dataset_id (str, optional) id of the dataset (if not fetched)
Returns:
list of Picture objects
add_data
Copy dataset.add_data(
name="myDataset",
version='myVersion',
pictures=[]
) Parameters:
name (string, optional) the name of the dataset
version (string, optional) the version name of the dataset to fetch if None, we'll take latest
pictures (list, required) The list of pictures to attach to add
Return:
None
delete
Delete the a dataset
Copy dataset.delete(
name="myDataset",
version='myVersion',
) Arguments:
name (string, optional) the name of the dataset to delete
version (string, optional) the version name of the dataset to delete if None, we'll take latest
Returns:
None
download
Download all the images from a dataset in a folder
Copy dataset.download(
dataset=None,
folder_name=None
) Arguments:
dataset (str, required) the name of the dataset you want to download written <dataset_name>/<version>
folder_name (str, optional) the name of the folder you want to download the pictures in, defaults to dataset_name/version if None
Returns:
None
add_annotation
Create an annotation for a picture in a dataset (or add objects to existing annotation)
Copy dataset.add_annotation(
picture_external_url=None
dataset_id=None,
data={},
image_qa={}
) Arguments:
picture_external_url (str, required) the name of the target image
dataset_id (str, optional) leave None if you already fetched a dataset
data (dict, required) annotation data
image_qa (dict, optional) Q&A data for image
Copy data = [
{
"type": "classification",
"label": "car"
}
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)
Copy data = [
{
"type": "rectangle",
"label": "car",
"rectangle": {
"top": 16,
"left": 10,
"width": 50,
"height": 60
}
}
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)
Copy data = [{
"type": "polygon",
"label" : "rose",
"polygon": {
"geometry": [
{
"x": 12,
"y": 15
},
{
"x": 178,
"y": 151
},
{
"x": 122,
"y": 196
},
{
"x": 112,
"y": 10
},
]
}
}
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)
Copy data = [
{
"type": "polygon",
"label" : "rose",
"polygon": {
"geometry": [
{
"x": 12,
"y": 15
},
{
"x": 178,
"y": 151
},
{
"x": 122,
"y": 196
},
{
"x": 112,
"y": 10
},
],
},
"qa": [{
"type": "text",
"question": "What color ?",
"answer": "red"
},
{
"type": "mc",
"question": "What color ?",
"answer": ["red"],
"choices": ["red", "yellow", "blue"]
},
{
"type": "select",
"question": "Is it raining ?",
"answer": "yes",
"choices": ["yes", "no"]
},
{
"type": "range",
"question": "size ?",
"answer": 68,
"max": 100,
"min": 0
}]
}
]
image_qa = [
{
"type": "text",
"question": "How much is the image rotated ?",
"answer": "approx. 32 deg."
},
{
"type": "mc",
"question": "image attribute",
"answer": ["high contrast"],
"choices": ["high contrast", "saturated"]
},
{
"type": "select",
"question": "image color",
"answer": "blue",
"choices": ["red", "blue", "green"]
},
{
"type": "range",
"question": "brigthness",
"answer": 36,
"max": 100,
"min": 0
},
]
ds.add_annotation("awsm_pic.jpg", True, data=data, image_qa=image_qa) Network
Networks are trained architectures that you can either deploy for inference (if available), use to start new experiments and share within your Organization's models.
The Network object
Copy {'organization': {'name': 'picsell'},
'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'tag': ['efficientdet', 'd2', 'COCO', 'base'],
'private': False,
'network_name': 'efficientdet-d2',
'description': 'This is a real game changer',
'model_object_name': '',
'checkpoint_object_name': '',
'origin_checkpoint_objects': {},
'type': 'detection',
'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.data-00000-of-00001',
'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.index'},
'thumb_object_name': 'b76ececa-274d-48de-b39e-70cf73941aba/effdet.png',
'framework': 'tensorflow2'
} Attributes
model_id (string) Unique identifier of your model
owner (hash, user_object) The creator of the model
network_name (string) The name of your model
description (string) A short description of what your model does
type (string) The type of application for your model, if you want to perform pre-annotation on Picsellia it has to be one of the following (but you can set your own type otherwise):
organization (hash, organization object) The organization under which your model is stored
private (boolean) Tells if your model is available for everyone in the public HUB or not
framework (string) The framework used for training
tag (list) List of tags to identify and sort your models
files (dict) Dictionary containing the list of files of your model
labels (dict) Dictionary of the labelmap of your model
base_parameters (dict) Dictionary of the base parameters allowing anyone to reproduce the training or iterate with already existing parameters
readme_text (str) A markdown text containing more information about your model
__init__
If you want to interact with your models, you have to initialize the Network class.
It is a subclass of the Client.
Copy from picsellia.client import Client
network = Client.Network(
api_token=None,
organization=None,
) Arguments:
api_token (string) Your personal API token
organization (string, optional) the name of the organization you want to work with (None default to your organization)
list
List of all the models for an organization
Returns:
A list containing the models of the chosen organization.
Copy [
{'organization': {'name': 'picsell'},
'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'tag': ['efficientdet', 'd2', 'COCO', 'base'],
'private': False,
'network_name': 'efficientdet-d2',
'description': 'This is a real game changer',
'model_object_name': '',
'checkpoint_object_name': '',
'origin_checkpoint_objects': {},
'type': 'detection',
'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.data-00000-of-00001',
'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.index'},
'thumb_object_name': 'b76ececa-274d-48de-b39e-70cf73941aba/effdet.png',
'framework': 'tensorflow2'
},
{'organization': {'name': 'picsell'},
'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'tag': None,
'private': True,
'network_name': 'vizdrone-test',
'description': 'This is a real game changer',
'model_object_name': '',
'checkpoint_object_name': '',
'origin_checkpoint_objects': {},
'type': 'detection',
'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-101.data-00000-of-00001',
'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-101.index'},
'thumb_object_name': '',
'framework': ''
}
] get
This method allows you to retrieve a particular model in order to update it or store some files.
Copy network.get(identifier=None) Arguments:
identifier (string) Either the name or the id of the model you want to retrieve
Returns:
The Network object
create
This methods allows you to create a new Network from the SDK
Copy network.create(
name=None,
type=None
) Attributes:
name (str, required) The name of your Network
type (str, required) the type of your Network, it is the task it performs such as :
'detection', for object detection
'segmentation', for object segmentation
'classification', for image classification
Returns:
Network object
update
This methods allows you to update the properties of a Network from your Organization
You must get or create a network before calling the update method.
Copy network.update(**kwargs) Arguments:
For example, if you want to update the description of your Network, you code will look like this :
Copy network.update(description="A brand new description for your Network") store
This method allows you to upload a file and save it under your model so you can use it later such as checkpoint files or config files.
Copy network.store(name="", path=None, zip=False) Arguments:
path (string, optional) the path to the file you want to upload
zip (boolean, optional) set to True if you want to zip your file or a folder to an archive before upload
update_thumb
This method allows you to set the image of your choice as the thumbnail displayed on the platform like below.
Copy network.update_thumb(path=None) Arguments:
path (string) the path to the image file
labels
This methods allows you to set the labelmap of your model.
Copy network.labels(labels=None) Arguments:
labels (dict) Dictionary containing your labelmap (index starts at 1 not 0)
Here is the COCO labelmap as example :
Copy labels = {
"1": "person",
"2": "bicycle",
"3": "car",
"4": "motorcycle",
"5": "airplane",
...
"86": "vase",
"87": "scissors",
"88": "teddy bear",
"89": "hair drier",
"90": "toothbrush"
} Returns:
None
Projects
The Project object
Copy {
"organization": {
"name": "pierre-nicolas"
},
"project_name": "test-logging",
"description": "No description",
"project_id": "3d039f84-7224-4442-b499-652f2f77f1a1",
"project_manager": {
"username": "pierre-nicolas"
},
"contributors": [
{
"user": {
"username": "pierre-nicolas"
},
"is_active": True,
"creation_date": "2021-01-18T12:08:51.573406Z"
}
],
"datasets": [
{
"dataset_name": "paginate",
"description": "",
"private": True
}
],
"type": "classification",
"creation_date": "2021-06-16"
} Attributes:
organization (hash) The parent organization of the project
project_name (string) The name of the project
description (string) A brief description of the project
project_id (UUID) The unique ID of the project
project_manager (hash) The user that created the project
contributors (hash) The list of worker objects for contributors of this project
datasets (hash) Short dataset objects for datasets attached to the project
type (string) Type of the project, you can set it arbitrarily
creation_date (string) Date of creation of this project
__init__
If you want to interact with your projects, you must initialize the Project class for our client.
Copy from picsellia.client import Client
clt = Client.Project(
api_token=None,
host="https://app.picsellia.com/sdk/",
project_token=None,
) Arguments:
api_token (string) Your personal API Token
host (string) The URL of Picsellia platform, you shouldn't have to change it
project_token (string) The token of the project you want work with (you can also let None and use get() or create() methods later)
Returns:
None
list
This method allows you to list every project in your organization.
Copy project_list = clt.list()
Copy Project Name: scans-file
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'public', 'description': None, 'private': True}, {'dataset_name': 'paginate', 'description': '', 'private': True}]
-------------
Project Name: scan-file2
Description: This is a real game changer
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: []
-------------
Project Name: test-logging
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'tttt', 'description': 'None', 'private': True}, {'dataset_name': 'paginate', 'description': '', 'private': True}]
-------------
Project Name: long-projet
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'confusion', 'description': None, 'private': True}]
-------------
... 10 more...
Project Name: Benchmark-sdk
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'cloned', 'description': 'None', 'private': True}]
-------------
Copy [{'contributors': [{'creation_date': '2021-01-18T12:08:51.573406Z',
'is_active': True,
'user': {'username': 'pierre-nicolas'}}],
'creation_date': '2021-04-20',
'datasets': [{'dataset_name': 'public',
'description': None,
'private': True},
{'dataset_name': 'paginate', 'description': '', 'private': True}],
'description': 'No description',
'organization': {'name': 'pierre-nicolas'},
'project_id': 'a0b90c90-3a7e-4dda-b6d0-59d6c4c19804',
'project_manager': {'username': 'pierre-nicolas'},
'project_name': 'scan-file',
'type': 'classification'},
{'contributors': [{'creation_date': '2021-01-18T12:08:51.573406Z',
'is_active': True,
'user': {'username': 'pierre-nicolas'}}],
'creation_date': '2021-07-27',
'datasets': [],
'description': 'This is a real game changer',
'organization': {'name': 'pierre-nicolas'},
'project_id': 'af76d4fe-3b8a-4587-a49f-d3bc69c19499',
'project_manager': {'username': 'pierre-nicolas'},
'project_name': 'scans-file2',
'type': 'classification'},
......
{'contributors': [{'creation_date': '2021-07-16T11:58:41.868208Z',
'is_active': False,
'user': {'username': 'pierre-nicolas'}},
{'creation_date': '2021-01-18T12:08:51.573406Z',
'is_active': True,
'user': {'username': 'pierre-nicolas'}}],
'creation_date': '2021-06-29',
'datasets': [{'dataset_name': 'cloned',
'description': 'None',
'private': True}],
'description': 'No description',
'organization': {'name': 'pierre-nicolas'},
'project_id': 'e89d03d0-77d9-4920-8899-202b1bb78cfd',
'project_manager': {'username': 'pierre-nicolas'},
'project_name': 'Benchmark-sdk',
'type': 'classification'}] get
This method allows you to retrieve a project so you can perform actions on it later.
Copy clt.get(
name="",
organization="null"
) Arguments:
name (string) The name of project you want to retrieve
organization (string) The name of the organization the project belongs to, leave "null" to stay in your organization
Returns:
self (The project object)
Copy project = clt.get(name='test-logging')
Copy Successfully fetched project test-logging
{
"organization": {
"name": "pierre-nicolas"
},
"project_name": "test-logging",
"description": "No description",
"project_id": "3d039f84-7224-4442-b499-652f2f77f1a1",
"project_manager": {
"username": "pierre-nicolas"
},
"contributors": [
{
"user": {
"username": "pierre-nicolas"
},
"is_active": True,
"creation_date": "2021-01-18T12:08:51.573406Z"
}
],
"datasets": [
{
"dataset_name": "paginate",
"description": "",
"private": True
}
],
"type": "classification",
"creation_date": "2021-06-16"
} create
This method allows you to create a new project.
You can't have 2 projects with the same name in your organization, you will have an error if the project already exists.
Copy clt.create(
name="",
organization="null",
**kwargs
) Arguments:
name (string, required) The name of the project you want to create
organization (string, optional) The organization where you want to create the project, leave "null" if you want it to be your organization
**kwargs , can be any attribute of the Project object within the following list :
Returns:
self (the Project object)
Example:
Here we will create a new project and specify the type in the **kwargs argument
Copy project = clt.create('new_project', type='detection')
Copy Project new_project2 created successfully.
{
"organization": {
"name": "pierre-nicolas"
},
"project_name": "new_project",
"description": "This is a real game changer",
"project_id": "e2b66355-f6b2-4db9-a6c9-75de43aea428",
"project_manager": {
"username": "pierre-nicolas"
},
"contributors": [
{
"user": {
"username": "pierre-nicolas"
},
"is_active": true,
"creation_date": "2021-01-18T12:08:51.573406Z"
}
],
"datasets": [],
"type": "detection",
"creation_date": "2021-07-27"
} As we can see, it did create a new project called 'new_project' and set the type to 'detection', easy no ?
update
This methods allows you to update some properties of your project
Copy clt.update(
project_name="",
description="",
type=""
) Arguments:
project_name (string, optional) the new name wanted for the project
description (string, optional) The description of the project
type (string, optional) The type of the project
Returns:
self (the Project object)
Copy project = clt.get('new_project')
project = project.update(
project_name='new_project_name',
description='a descriptive text'
)
Copy Project new_project_name updated.
{
"organization": {
"name": "pierre-nicolas"
},
"project_name": "new_project_name",
"description": "a descriptive text",
"project_id": "49fc1dc9-0eb9-4aa4-ad40-0c30be3868e9",
"project_manager": {
"username": "pierre-nicolas"
},
"contributors": [
{
"user": {
"username": "pierre-nicolas"
},
"is_active": True,
"creation_date": "2021-01-18T12:08:51.573406Z"
}
],
"datasets": [],
"type": "classification",
"creation_date": "2021-07-27"
} As we can see, we have successfully updated the name and the description of our project !
delete
This method allows you to delete a project.
Returns:
None
Copy project = clt.get('new_project')
project.delete()
Copy Project new_project deleted. attach_dataset
This method allows you to attach a dataset from your organization to your project, just like you would do in the platform.
Copy clt.attach_dataset(
dataset=""
) Arguments:
dataset (string, required) The dataset name and version separated by as '/' : <dataset_name>/<version>
Returns:
None
Copy project = clt.get('new_project')
project.attach_dataset('Tutorial/first')
Copy Dataset Tutorial/first successfully attached to project new_project Experiments
The experiment object
Copy {
'id': 'b8995c82-f304-40d2-a91f-7e2bcc5df429',
'date_created': '2020-12-06T20:09:29.703959Z',
'last_update': '2020-12-06T20:09:29.703689Z',
'owner': {'username': 'Pierre-Nicolas'},
'project': {
'project_id': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'project_name': 'project 21'
},
'name': 'exp2',
'description': '',
'status': 'started',
'logging': None,
'files': [
{
'id': 13,
'date_created': '2020-12-08T23:31:13.389276Z',
'last_update': '2020-12-08T23:31:13.388914Z',
'large': True,
'name': 'model-latest',
'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/saved_model.zip'
},
{
'id': 14,
'date_created': '2020-12-08T23:35:42.964512Z',
'last_update': '2020-12-08T23:35:42.964274Z',
'large': False,
'name': 'config',
'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/pipeline.config'
}
],
'data': [
{
'id': 6,
'date_created': '2020-12-07T13:14:15.846681Z',
'last_update': '2020-12-07T13:14:15.846467Z',
'type': 'value'
'name': 'acc',
'data': 0.8
},
]
}
Attributes:
id (string) Unique identifier for the object
owner (hash, user object) The creator of the object
name (string) The name of the experiment
description (string) A short description of the experiment
status (string) Describes the experiment state
date_created (date) Date and time at which the object was created
last_update (date) Date and time of the last update of the object
logging (hash, logging object) The telemetry of the experiment
__init__
If you need to interact with your experiments, you must initialize the Experiment class.
It is a subclass of the Client.
It is also used to initialize some class parameters such as project_token, id (experiment id) or name (experiment name), so you don't have to specify them when you call a method of the Experiment class.
Please note that when calling an Experiment method, you can either specify the id or the name of the experiment, you don't have to enter both .
Copy from picsellia.client import Client
experiment = Client.Experiment(
api_token=None,
host="https://beta.picsellia.com/sdk/",
project_token=None,
id=None,
name=None,
interactive=True
) Arguments:
api_token (string) Your personal API token
host (string, optional) the URL of Picsell.ia platform, you shouldn't have to change it
project_token (string, optional) the token of the project you want to focus on
id (string, optional) the id of the experiment you want to work with
name (string, optional) the name of the experiment you want to work with
interactive (boolean, optional) if set to False, disable all prompt that requires user action
Returns:
None
checkout
Copy experiment.checkout(
id=None,
name=None,
project_token=None,
tree=False,
with_file=False,
with_data=False
) Arguments
id (string, optional) the id of the experiment you want to work with
name (string, optional) the name of the experiment you want to work with
project_token (string, optional) the token of the project you want to focus on
with_files (boolean, optional), set to True if you want to download all the files of your experiment
with_data (boolean, optional), set to True if you want to retrieve all data assets from your experiment
Returns:
JSON-like object
Copy {
'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'date_created': '2020-12-12T16:26:59.343322Z',
'last_update': '2020-12-12T16:26:59.343090Z',
'owner': 1,
'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'name': 'my_new_experiment',
'description': 'dog cat classificaiton with base parameters',
'status': 'started',
'logging': None,
'files': [],
'data': []
} list
List all of the experiments related to one project
Copy experiment.list(project_token=None) Arguments:
project_token (string, optional) the the token of the project you want to focus
Returns:
A list containing the experiment objects for your project.
Copy [
{'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'date_created': '2020-12-12T16:26:59.343322Z',
'last_update': '2020-12-12T16:26:59.343090Z',
'owner': {'username': 'Pierre-Nicolas'},
'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'name': 'my_new_experiment',
'description': 'dog cat classificaiton with base parameters',
'status': 'started',
'logging': None,
'files': [],
'data': []},
{'experiment_id': 'b251ca51-ba03-4144-a1e7-9eb15db46984',
'date_created': '2020-12-06T20:10:19.101950Z',
'last_update': '2020-12-06T20:10:19.101772Z',
'owner': {'username': 'Pierre-Nicolas'},
'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'name': 'desc',
'description': '',
'status': 'started',
'logging': None,
'files': [],
'data': []}
] create
Creates a new experiment for a project
Copy experiment.create(
name=None,
description='',
previous=None,
dataset=None,
source=None,
) Parameters:
name (string, 60 char max.) the name of your new experiment
description (string, 1000 char max., optional) a short description for your experiment
previous (string, optional) set it to a previous experiment name if you want to iterate
dataset (string, optional) set it to <dataset_name>/<version> of the dataset you want to use for your experiment
source (string, optional) set it to <username>/<model_name> to clone the assets of a model from our model HUB or your organization
Returns:
An experiment object with your created experiment
Copy {
'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'date_created': '2020-12-12T16:26:59.343322Z',
'last_update': '2020-12-12T16:26:59.343090Z',
'owner': 1,
'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'name': 'my_new_experiment',
'description': 'dog cat classificaiton with base parameters',
'status': 'started',
'logging': None,
'files': [],
'data': []
} update
From here, in order to run all the following methods, you mst have checkout or created an experiment first.
This methods allows you to update the information of an experiment
Copy experiment.update(
**kwargs
) Arguments:
**kwargs
name (string) the new name for your experiment
description (string) the new description
status (string) the new status
Returns:
The updated experiment object
Copy
{'id': 'b76ececa-274d-48de-b39e-70cf73941aba',
'date_created': '2020-12-12T16:26:59.343322Z',
'last_update': '2020-12-12T16:26:59.343090Z',
'owner': 1,
'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
'name': 'updated_name',
'description': 'dog cat classificaiton with base parameters',
'status': 'started',
'logging': None,
'files': [],
'data': []} delete
This method allows you to delete one experiment
Returns:
True if the experiment has been deleted correctly
delete_all
Delete all the experiments for a given project
Copy experiment.delete_all(project_token=None) Arguments:
project_token (uuid, optional) the project token of the project you want to delete all experiments
Returns:
True if the experiments has been deleted correctly
dl_annotations
Download the annotations JSON for the dataset attached to an experiment
Copy experiment.dl_annotations(type='all') Arguments:
type (string, required)
'all' (default) download all annotations of the dataset
'accepted' download only accepted annotations from the dataset
Returns:
JSON containing all of the annotations in the COCO format, the JSON is also stored in experiment.dict_annotations
File-assets
The file-asset object
Copy {
'id': 13,
'date_created': '2020-12-08T23:31:13.389276Z',
'last_update': '2020-12-08T23:31:13.388914Z',
'large': True,
'name': 'model-latest',
'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/saved_model.zip'
} Attributes
id (int) Unique identifier for the object
name (string) The name of the file-asset
object_name (string) The path to the file in our object storage
date_created (date) Date and time at which the object was created
last_update (date) Date and time of the last update of the object
store
Copy experiment.store(name="", path=None, zip=False) Arguments:
name (string,) the name of the file, it will be displayed on Picsell.ia
path (string, optional) the path to the file you want to upload
zip (boolean, optional) set to True if you want to zip your file or a folder to an archive before upload
list_files
List all the files belonging to one experiment
Copy experiment.list_files() Returns:
A list of all the file asset object of your experiment
Copy [{'id': 22,
'date_created': '2021-02-09T12:32:18.022694Z',
'last_update': '2021-02-09T12:32:18.022465Z',
'large': False,
'name': 'config',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'},
{'id': 23,
'date_created': '2021-02-09T12:32:18.068900Z',
'last_update': '2021-02-09T12:32:18.068723Z',
'large': True,
'name': 'model-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/0/saved_model.zip'},
{'id': 24,
'date_created': '2021-02-09T12:32:18.112582Z',
'last_update': '2021-02-09T12:32:18.112410Z',
'large': True,
'name': 'checkpoint-data-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.data-00000-of-00001'},
{'id': 25,
'date_created': '2021-02-09T12:32:18.156945Z',
'last_update': '2021-02-09T12:32:18.156767Z',
'large': False,
'name': 'checkpoint-index-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.index'}] delete_all_files
Delete all the files belonging to one experiment
Copy experiment.delete_all_files() Returns:
True if all the files has been deleted correctly
get_file
Retrieve one particular file of an experiment by its name
Copy experiment.get_file(name=None) Arguments:
name (string) the name of the file asset you want to retrieve
Returns:
A file-asset object
Copy {'id': 22,
'date_created': '2021-02-09T12:32:18.022694Z',
'last_update': '2021-02-09T12:32:18.022465Z',
'large': False,
'name': 'config',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'} delete_file
Delete a file from your experiment
Copy experiment.delete_file(name=None) Arguments:
name (string) the name of the file asset you want to delete
Returns:
True if the file has been deleted correctly
update_file
Update the information of a given file
Copy experiment.update_file(file_name, **kwargs) Arguments:
file_name (string) the name of the file asset you want to update
**kwargs
name (string) the new name for your file
Returns:
A file-asset containing the updated object
Data-assets
The data-asset object
Copy {
'id': 72,
'date_created': '2021-02-09T12:32:18.293746Z',
'last_update': '2021-02-09T12:32:18.293556Z',
'name': 'parameters',
'data': {'steps': 200000,
'nb_gpu': 1,
'batch_size': 8,
'learning_rate': 0.005,
'annotation_type': 'classification'
},
'type': 'table'
} Attributes
id (int) Unique identifier for the object
name (string) The name of the data-asset
data (hash, data object) The data logged in this object
date_created (date) Date and time at which the object was created
last_update (date) Date and time of the last update of the object
log
Copy experiment.log(name=None, data={}, type="") Arguments:
name (string) the name of the data-asset you want to create
data (dict) a dictionary containing the data you want to display on the platform
list_data
List all the data-assets belonging to one experiment
Copy experiment.list_data() Returns:
A list of all the data-assets object of your experiment
Copy [
{
'id': 72,
'date_created': '2021-02-09T12:32:18.293746Z',
'last_update': '2021-02-09T12:32:18.293556Z',
'name': 'parameters',
'data': {'steps': 200000,
'nb_gpu': 1,
'batch_size': 8,
'learning_rate': 0.005,
'annotation_type': 'classification'
},
'type': 'table'
}
] delete_all_data
Delete all the files belonging to one experiment
Copy experiment.delete_all_data() Returns:
True if all the data-assets has been deleted correctly
get_data
Retrieve one particular file of an experiment by its name
Copy experiment.get_data(name=None) Arguments:
name (string) the name of the data-asset you want to retrieve
Returns:
A data-asset object
Copy {'steps': 200000,
'nb_gpu': 1,
'batch_size': 8,
'learning_rate': 0.005,
'annotation_type': 'classification'} delete_data
Delete a file from your experiment
Copy experiment.delete_data(name=None) name (string) the name of the data-asset you want to delete
Returns:
True if the data-asset has been deleted correctly
update_data
Update the information of a given file
Copy experiment.update_data(data_name, **kwargs) Arguments:
data_name (string) the name of the data-asset you want to update
**kwargs
name (string) the new name for your data-asset
Returns:
A data-asset containing the updated object
Copy {
'id': 72,
'date_created': '2021-02-09T12:32:18.293746Z',
'last_update': '2021-02-09T12:32:18.293556Z',
'name': 'parameters',
'data': {'steps': 5000000.0,
'nb_gpu': 8,
'batch_size': 64,
'learning_rate': 0.0055,
'annotation_type': 'detection'},
'type': 'table'
} Logging (coming soon)
Logging is the telemetry (if existing) of your experiment. It allows you to record in real time what's happening in the interpreter during an experiment and to monitor it from the platform.
The logging object
Attributes
logs (dict) The raw stdout of your interpreter
steps (dict) The categories of your logs, useful to navigate inside your logs
timestamps (dict) The timestamps for each step
exit_code (dict) The exit code and the timestamp of the end of your run
