fedbiomed.common.repository

Module: fedbiomed.common.repository

HTTP file repository from which to upload and download files.

Classes

Repository

CLASS 
Repository(uploads_url, tmp_dir, cache_dir)

HTTP file repository from which to upload and download files.

Files are uploaded from/downloaded to a temporary file (tmp_dir). Data uploaded should be:

  • python code (*.py file) that describes model + data handling/preprocessing
  • model params (under *.pt format)

Parameters:

Name Type Description Default
uploads_url Union[Text, bytes]

The URL where we upload files

 required
tmp_dir str

A directory for temporary files

 required
cache_dir str

Currently unused

 required
Source code in fedbiomed/common/repository.py 
def __init__(self,
             uploads_url: Union[Text, bytes],
             tmp_dir: str,
             cache_dir: str):
    """Constructor of the class.

    Args:
        uploads_url: The URL where we upload files
        tmp_dir: A directory for temporary files
        cache_dir: Currently unused
    """

    self.uploads_url = uploads_url
    self.tmp_dir = tmp_dir
    self.cache_dir = cache_dir  # unused

Attributes

cache_dir instance-attribute 
cache_dir = cache_dir
tmp_dir instance-attribute 
tmp_dir = tmp_dir
uploads_url instance-attribute 
uploads_url = uploads_url

Functions

download_file(url, filename)

Downloads a file from a HTTP file repository (through an HTTP GET request).

Parameters:

Name Type Description Default
url str

An url from which to download file

 required
filename str

The name of the temporary file

 required

Returns:

Name Type Description
status int

The HTTP status code
filepath str

The complete pathfile under which the temporary file is saved
Source code in fedbiomed/common/repository.py 
def download_file(self, url: str, filename: str) -> Tuple[int, str]:
    """Downloads a file from a HTTP file repository (through an HTTP GET request).

    Args:
        url: An url from which to download file
        filename: The name of the temporary file

    Returns:
        status: The HTTP status code
        filepath: The complete pathfile under which the temporary file is saved
    """

    res = self._request_handler(requests.get, url, filename)
    self._raise_for_status_handler(res, filename)
    filepath = os.path.join(self.tmp_dir, filename)

    try:
        open(filepath, 'wb').write(res.content)
    except FileNotFoundError as err:
        _msg = ErrorNumbers.FB604.value + str(err) + ', cannot save the downloaded content into it'
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    except PermissionError:
        _msg = ErrorNumbers.FB604.value + f': Unable to read {filepath} due to unsatisfactory privileges'
        ", cannot write the downloaded content into it"
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    except MemoryError:
        _msg = ErrorNumbers.FB604.value + f" : cannot write on {filepath}: out of memory!"
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    except OSError:
        _msg = ErrorNumbers.FB604.value + f': Cannot open file {filepath} after downloading'
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)

    return res.status_code, filepath

upload_file(filename)

Uploads a file to an HTTP file repository (through an HTTP POST request).

Parameters:

Name Type Description Default
filename str

A name/path of the file to upload.

 required

Returns:

Type Description
Dict[str, Any]

The result of the request under JSON format.

Raises:

Type Description
FedbiomedRepositoryError

unable to read the file 'filename'
FedbiomedRepositoryError

POST HTTP request fails or returns an HTTP status 4xx (bad request) or 500 (internal server error)
FedbiomedRepositoryError

unable to deserialize JSON from the request
Source code in fedbiomed/common/repository.py 
def upload_file(self, filename: str) -> Dict[str, Any]:
    """Uploads a file to an HTTP file repository (through an HTTP POST request).

    Args:
        filename: A name/path of the file to upload.

    Returns:
        The result of the request under JSON format.

    Raises:
        FedbiomedRepositoryError: unable to read the file 'filename'
        FedbiomedRepositoryError: POST HTTP request fails or returns
            an HTTP status 4xx (bad request) or 500 (internal server error)
        FedbiomedRepositoryError: unable to deserialize JSON from
            the request
    """
    # first, we are trying to open the file `filename` and catch
    # any known exceptions related top `open` builtin function
    try:
        files = {'file': open(filename, 'rb')}
    except FileNotFoundError:
        _msg = ErrorNumbers.FB604.value + f': File {filename} not found, cannot upload it'
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    except PermissionError:
        _msg = ErrorNumbers.FB604.value + f': Unable to read {filename} due to unsatisfactory privileges' + \
            ", cannot upload it"
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    except OSError:
        _msg = ErrorNumbers.FB604.value + f': Cannot read file {filename} when uploading'
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)

    # second, we are issuing an HTTP 'POST' request to the HTTP server

    _res = self._request_handler(requests.post, self.uploads_url,
                                 filename, files=files)
    # checking status of HTTP request

    self._raise_for_status_handler(_res, filename)

    # finally, we are deserializing message from JSON
    try:
        json_res = _res.json()
    except JSONDecodeError:
        # might be triggered by `request` package when deserializing
        _msg = 'Unable to deserialize JSON from HTTP POST request (when uploading file)'
        logger.error(_msg)
        raise FedbiomedRepositoryError(_msg)
    return json_res