core.internals.filesystems¶

Submodule of khiops.core.internals

Classes to interact with local and remote filesystems

Functions¶

`child_path`	Creates a path with a child appended
`child_uri_info`	Creates a URI info with a path with child appended
`copy_from_local`	Copies the content of a local file to the resource
`copy_to_local`	Copies the content of the resource to a local path
`create_resource`	Factory method to create a FilesystemResource from an URI
`exists`	Checks the existence of the resource
`get_child_path`	Returns the child path of this URI at the specified child name
`get_parent_path`	Returns the specified parent path of this URI
`is_local_resource`	Checks if a URI or path is effectively a local path
`list_dir`	Lists a directory resource contents
`make_dir`	Creates a directory at the resource's path
`parent_path`	Returns the parent of the specified path
`parent_uri_info`	Creates the parent for the input URI info
`read`	Reads data in binary mode from the target resource
`remove`	Removes the resource
`write`	Writes data in binary mode to the target resource

Classes¶

`AmazonS3Resource`	Amazon Simple Storage Service (S3) Resource
`FilesystemResource`	Abstract Filesystem Resource
`GoogleCloudStorageResource`	Google Cloud Storage Resource
`LocalFilesystemResource`	A local filesystem resource

class khiops.core.internals.filesystems.AmazonS3Resource(uri)¶

Bases: FilesystemResource

Amazon Simple Storage Service (S3) Resource

The default configuration and credentials are read from the paths

~/.aws/configuration
~/.aws/credentials

The location of the configuration and credentials files may be overridden using the following environment variables:

AWS_CONFIG_FILE: location of the configuration file
AWS_SHARED_CREDENTIALS_FILE: location of the credentials file

If no configuration/credentials files are usable, Amazon SDK defaults apply. Individual settings such as endpoint URL or region can be used to override any of the available settings.

Other relevant environment variables:

- AWS_S3_ENDPOINT_URL: sets the service endpoint URL
- AWS_DEFAULT_REGION: sets the region to send requests to

Note

Operations with the s3 client are only verified by checking that the HTTP response code is in the 200 range.

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.FilesystemResource(uri)¶

Bases: ABC

Abstract Filesystem Resource

abstract create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

abstract create_parent(file_name)¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.GoogleCloudStorageResource(uri)¶

Bases: FilesystemResource

Google Cloud Storage Resource

By default it reads the configuration from standard location.

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.LocalFilesystemResource(uri)¶

Bases: FilesystemResource

A local filesystem resource

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

khiops.core.internals.filesystems.child_path(path, child_name)¶

Creates a path with a child appended

Notes

This function always return a posix path (“/” as separator). For example for the windows path and child name:

parent: C:\Program Files
child:  khiops

this method returns::: C:/Program Files/khiops

khiops.core.internals.filesystems.child_uri_info(uri_info, child_name)¶

Creates a URI info with a path with child appended

Parameters:

uri_infourllib.parse.ParseResult: URI info structure (output of urllib.parse.urlparse)
child_namestr: Name of the new childe node

Returns:

urllib.parse.ParseResult: URI info structure for the child URI

khiops.core.internals.filesystems.copy_from_local(uri_or_path, local_path)¶

Copies the content of a local file to the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
local_pathstr: The path of the local file to be copied.

Raises:

RuntimeError: If there was a problem when removing.

khiops.core.internals.filesystems.copy_to_local(uri_or_path, local_path)¶

Copies the content of the resource to a local path

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
local_pathstr: The path of the local file to be copied.

Raises:

RuntimeError: If there was a problem when removing.

khiops.core.internals.filesystems.create_resource(uri_or_path)¶

Factory method to create a FilesystemResource from an URI

Parameters:

uri_or_pathstr

The resource’s URI . Supported protocols/schemes:

file or empty: Local filesystem resource
s3: Amazon S3 resource
gs: Google Cloud Storage resource

Returns:

FilesystemResource: The URI resource object, its class depends on the URI.

khiops.core.internals.filesystems.exists(uri_or_path)¶

Checks the existence of the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

bool: True if the resource exists.

khiops.core.internals.filesystems.get_child_path(uri_or_path, child_name)¶

Returns the child path of this URI at the specified child name

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
child_namestr: Name of the child to be added to the URI or path.

Returns:

str: The URI or path of the child resource.

khiops.core.internals.filesystems.get_parent_path(uri_or_path)¶

Returns the specified parent path of this URI

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

str: The URI or path of the parent’s resource.

khiops.core.internals.filesystems.is_local_resource(uri_or_path)¶

Checks if a URI or path is effectively a local path

Note

An URI with scheme of size 1 will be considered a local path. This is to take into account Windows paths such as C:\Some\Windows\Path.

Returns:

bool: True if a URI refers to a local path

khiops.core.internals.filesystems.list_dir(uri_or_path)¶

Lists a directory resource contents

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

list: A list of paths or URIs.

khiops.core.internals.filesystems.make_dir(uri_or_path)¶

Creates a directory at the resource’s path

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

khiops.core.internals.filesystems.parent_path(path)¶

Returns the parent of the specified path

Notes

This function always return a posix path (“/” as separator). For example for the windows path:

C:\Program Files\khiops

this method returns:

C:/Program Files

khiops.core.internals.filesystems.parent_uri_info(uri_info)¶

Creates the parent for the input URI info

Parameters:

uri_infourllib.parse.ParseResult: URI info structure (output of urllib.parse.urlparse)

Returns:

urllib.parse.ParseResult: URI info structure for the parent URI

khiops.core.internals.filesystems.read(uri_or_path, size=None)¶

Reads data in binary mode from the target resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
sizeint, optional: Number of bytes to read.

Returns:

bytes: A buffer containing the read contents.

Raises:

RuntimeError: If there was a problem when reading.

khiops.core.internals.filesystems.remove(uri_or_path)¶

Removes the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Raises:

RuntimeError: If there was a problem when removing.

khiops.core.internals.filesystems.write(uri_or_path, data)¶

Writes data in binary mode to the target resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
datastr or bytes: The data to be written.

Raises:

RuntimeError: If there was a problem when writing.