core.internals.filesystems¶
Submodule of khiops.core.internals
Classes to interact with local and remote filesystems
Functions¶
Creates a path with a child appended |
|
Creates a URI info with a path with child appended |
|
Copies the content of a local file to the resource |
|
Copies the content of the resource to a local path |
|
Factory method to create a FilesystemResource from an URI |
|
Checks the existence of the resource |
|
Returns the child path of this URI at the specified child name |
|
Returns the specified parent path of this URI |
|
Checks if a URI or path is effectively a local path |
|
Lists a directory resource contents |
|
Creates a directory at the resource's path |
|
Returns the parent of the specified path |
|
Creates the parent for the input URI info |
|
Reads data in binary mode from the target resource |
|
Removes the resource |
|
Writes data in binary mode to the target resource |
Classes¶
Amazon Simple Storage Service (S3) Resource |
|
Abstract Filesystem Resource |
|
Google Cloud Storage Resource |
|
A local filesystem resource |
- class khiops.core.internals.filesystems.AmazonS3Resource(uri)¶
Bases:
FilesystemResource
Amazon Simple Storage Service (S3) Resource
The default configuration and credentials are read from the paths
~/.aws/configuration
~/.aws/credentials
The location of the configuration and credentials files may be overridden using the following environment variables:
AWS_CONFIG_FILE
: location of the configuration fileAWS_SHARED_CREDENTIALS_FILE
: location of the credentials file
If no configuration/credentials files are usable, Amazon SDK defaults apply. Individual settings such as endpoint URL or region can be used to override any of the available settings.
Other relevant environment variables:
- AWS_S3_ENDPOINT_URL: sets the service endpoint URL - AWS_DEFAULT_REGION: sets the region to send requests to
Note
Operations with the s3 client are only verified by checking that the HTTP response code is in the 200 range.
- create_child(file_name)¶
Creates a resource representing a child of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- create_parent()¶
Creates a resource representing the parent of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- class khiops.core.internals.filesystems.FilesystemResource(uri)¶
Bases:
ABC
Abstract Filesystem Resource
- abstract create_child(file_name)¶
Creates a resource representing a child of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- abstract create_parent(file_name)¶
Creates a resource representing the parent of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- class khiops.core.internals.filesystems.GoogleCloudStorageResource(uri)¶
Bases:
FilesystemResource
Google Cloud Storage Resource
By default it reads the configuration from standard location.
- create_child(file_name)¶
Creates a resource representing a child of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- create_parent()¶
Creates a resource representing the parent of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- class khiops.core.internals.filesystems.LocalFilesystemResource(uri)¶
Bases:
FilesystemResource
A local filesystem resource
- create_child(file_name)¶
Creates a resource representing a child of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- create_parent()¶
Creates a resource representing the parent of this instance
- Returns:
FilesystemResource
The specific resource type is that of the caller
- khiops.core.internals.filesystems.child_path(path, child_name)¶
Creates a path with a child appended
Notes
This function always return a posix path (“/” as separator). For example for the windows path and child name:
parent: C:\Program Files child: khiops
- this method returns::
C:/Program Files/khiops
- khiops.core.internals.filesystems.child_uri_info(uri_info, child_name)¶
Creates a URI info with a path with child appended
- Parameters:
- uri_info
urllib.parse.ParseResult
URI info structure (output of
urllib.parse.urlparse
)- child_namestr
Name of the new childe node
- uri_info
- Returns:
urllib.parse.ParseResult
URI info structure for the child URI
- khiops.core.internals.filesystems.copy_from_local(uri_or_path, local_path)¶
Copies the content of a local file to the resource
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- local_pathstr
The path of the local file to be copied.
- Raises:
- RuntimeError
If there was a problem when removing.
- khiops.core.internals.filesystems.copy_to_local(uri_or_path, local_path)¶
Copies the content of the resource to a local path
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- local_pathstr
The path of the local file to be copied.
- Raises:
- RuntimeError
If there was a problem when removing.
- khiops.core.internals.filesystems.create_resource(uri_or_path)¶
Factory method to create a FilesystemResource from an URI
- Parameters:
- uri_or_pathstr
The resource’s URI . Supported protocols/schemes:
file
or empty: Local filesystem resources3
: Amazon S3 resourcegs
: Google Cloud Storage resource
- Returns:
FilesystemResource
The URI resource object, its class depends on the URI.
- khiops.core.internals.filesystems.exists(uri_or_path)¶
Checks the existence of the resource
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- Returns:
bool
True if the resource exists.
- khiops.core.internals.filesystems.get_child_path(uri_or_path, child_name)¶
Returns the child path of this URI at the specified child name
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- child_namestr
Name of the child to be added to the URI or path.
- Returns:
- str
The URI or path of the child resource.
- khiops.core.internals.filesystems.get_parent_path(uri_or_path)¶
Returns the specified parent path of this URI
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- Returns:
- str
The URI or path of the parent’s resource.
- khiops.core.internals.filesystems.is_local_resource(uri_or_path)¶
Checks if a URI or path is effectively a local path
Note
An URI with scheme of size 1 will be considered a local path. This is to take into account Windows paths such as
C:\Some\Windows\Path
.
- khiops.core.internals.filesystems.list_dir(uri_or_path)¶
Lists a directory resource contents
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- Returns:
- list
A list of paths or URIs.
- khiops.core.internals.filesystems.make_dir(uri_or_path)¶
Creates a directory at the resource’s path
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- khiops.core.internals.filesystems.parent_path(path)¶
Returns the parent of the specified path
Notes
This function always return a posix path (“/” as separator). For example for the windows path:
C:\Program Files\khiops
this method returns:
C:/Program Files
- khiops.core.internals.filesystems.parent_uri_info(uri_info)¶
Creates the parent for the input URI info
- Parameters:
- uri_info
urllib.parse.ParseResult
URI info structure (output of
urllib.parse.urlparse
)
- uri_info
- Returns:
urllib.parse.ParseResult
URI info structure for the parent URI
- khiops.core.internals.filesystems.read(uri_or_path, size=None)¶
Reads data in binary mode from the target resource
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- sizeint, optional
Number of bytes to read.
- Returns:
bytes
A buffer containing the read contents.
- Raises:
- RuntimeError
If there was a problem when reading.
- khiops.core.internals.filesystems.remove(uri_or_path)¶
Removes the resource
- Parameters:
- uri_or_pathstr
The resource’s URI or local filesystem path.
- Raises:
- RuntimeError
If there was a problem when removing.