core.internals.filesystems

Submodule of khiops.core.internals

Classes to interact with local and remote filesystems

Functions

child_path

Creates a path with a child appended

child_uri_info

Creates a URI info with a path with child appended

copy_from_local

Copies the content of a local file to the resource

copy_to_local

Copies the content of the resource to a local path

create_resource

Factory method to create a FilesystemResource from an URI

exists

Checks the existence of the resource

get_child_path

Returns the child path of this URI at the specified child name

get_parent_path

Returns the specified parent path of this URI

is_local_resource

Checks if a URI or path is effectively a local path

list_dir

Lists a directory resource contents

make_dir

Creates a directory at the resource's path

parent_path

Returns the parent of the specified path

parent_uri_info

Creates the parent for the input URI info

read

Reads data in binary mode from the target resource

remove

Removes the resource

write

Writes data in binary mode to the target resource

Classes

AmazonS3Resource

Amazon Simple Storage Service (S3) Resource

FilesystemResource

Abstract Filesystem Resource

GoogleCloudStorageResource

Google Cloud Storage Resource

LocalFilesystemResource

A local filesystem resource

class khiops.core.internals.filesystems.AmazonS3Resource(uri)

Bases: FilesystemResource

Amazon Simple Storage Service (S3) Resource

The default configuration and credentials are read from the paths

  • ~/.aws/configuration

  • ~/.aws/credentials

The location of the configuration and credentials files may be overridden using the following environment variables:

  • AWS_CONFIG_FILE: location of the configuration file

  • AWS_SHARED_CREDENTIALS_FILE: location of the credentials file

If no configuration/credentials files are usable, Amazon SDK defaults apply. Individual settings such as endpoint URL or region can be used to override any of the available settings.

Other relevant environment variables:

- AWS_S3_ENDPOINT_URL: sets the service endpoint URL
- AWS_DEFAULT_REGION: sets the region to send requests to

Note

Operations with the s3 client are only verified by checking that the HTTP response code is in the 200 range.

create_child(file_name)

Creates a resource representing a child of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

create_parent()

Creates a resource representing the parent of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

class khiops.core.internals.filesystems.FilesystemResource(uri)

Bases: ABC

Abstract Filesystem Resource

abstract create_child(file_name)

Creates a resource representing a child of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

abstract create_parent(file_name)

Creates a resource representing the parent of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

class khiops.core.internals.filesystems.GoogleCloudStorageResource(uri)

Bases: FilesystemResource

Google Cloud Storage Resource

By default it reads the configuration from standard location.

create_child(file_name)

Creates a resource representing a child of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

create_parent()

Creates a resource representing the parent of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

class khiops.core.internals.filesystems.LocalFilesystemResource(uri)

Bases: FilesystemResource

A local filesystem resource

create_child(file_name)

Creates a resource representing a child of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

create_parent()

Creates a resource representing the parent of this instance

Returns:
FilesystemResource

The specific resource type is that of the caller

khiops.core.internals.filesystems.child_path(path, child_name)

Creates a path with a child appended

Notes

This function always return a posix path (“/” as separator). For example for the windows path and child name:

parent: C:\Program Files
child:  khiops
this method returns::

C:/Program Files/khiops

khiops.core.internals.filesystems.child_uri_info(uri_info, child_name)

Creates a URI info with a path with child appended

Parameters:
uri_infourllib.parse.ParseResult

URI info structure (output of urllib.parse.urlparse)

child_namestr

Name of the new childe node

Returns:
urllib.parse.ParseResult

URI info structure for the child URI

khiops.core.internals.filesystems.copy_from_local(uri_or_path, local_path)

Copies the content of a local file to the resource

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

local_pathstr

The path of the local file to be copied.

Raises:
RuntimeError

If there was a problem when removing.

khiops.core.internals.filesystems.copy_to_local(uri_or_path, local_path)

Copies the content of the resource to a local path

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

local_pathstr

The path of the local file to be copied.

Raises:
RuntimeError

If there was a problem when removing.

khiops.core.internals.filesystems.create_resource(uri_or_path)

Factory method to create a FilesystemResource from an URI

Parameters:
uri_or_pathstr

The resource’s URI . Supported protocols/schemes:

  • file or empty: Local filesystem resource

  • s3: Amazon S3 resource

  • gs: Google Cloud Storage resource

Returns:
FilesystemResource

The URI resource object, its class depends on the URI.

khiops.core.internals.filesystems.exists(uri_or_path)

Checks the existence of the resource

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

Returns:
bool

True if the resource exists.

khiops.core.internals.filesystems.get_child_path(uri_or_path, child_name)

Returns the child path of this URI at the specified child name

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

child_namestr

Name of the child to be added to the URI or path.

Returns:
str

The URI or path of the child resource.

khiops.core.internals.filesystems.get_parent_path(uri_or_path)

Returns the specified parent path of this URI

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

Returns:
str

The URI or path of the parent’s resource.

khiops.core.internals.filesystems.is_local_resource(uri_or_path)

Checks if a URI or path is effectively a local path

Note

An URI with scheme of size 1 will be considered a local path. This is to take into account Windows paths such as C:\Some\Windows\Path.

Returns:
bool

True if a URI refers to a local path

khiops.core.internals.filesystems.list_dir(uri_or_path)

Lists a directory resource contents

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

Returns:
list

A list of paths or URIs.

khiops.core.internals.filesystems.make_dir(uri_or_path)

Creates a directory at the resource’s path

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

khiops.core.internals.filesystems.parent_path(path)

Returns the parent of the specified path

Notes

This function always return a posix path (“/” as separator). For example for the windows path:

C:\Program Files\khiops

this method returns:

C:/Program Files
khiops.core.internals.filesystems.parent_uri_info(uri_info)

Creates the parent for the input URI info

Parameters:
uri_infourllib.parse.ParseResult

URI info structure (output of urllib.parse.urlparse)

Returns:
urllib.parse.ParseResult

URI info structure for the parent URI

khiops.core.internals.filesystems.read(uri_or_path, size=None)

Reads data in binary mode from the target resource

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

sizeint, optional

Number of bytes to read.

Returns:
bytes

A buffer containing the read contents.

Raises:
RuntimeError

If there was a problem when reading.

khiops.core.internals.filesystems.remove(uri_or_path)

Removes the resource

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

Raises:
RuntimeError

If there was a problem when removing.

khiops.core.internals.filesystems.write(uri_or_path, data)

Writes data in binary mode to the target resource

Parameters:
uri_or_pathstr

The resource’s URI or local filesystem path.

datastr or bytes

The data to be written.

Raises:
RuntimeError

If there was a problem when writing.