---
name: megfile
description: Use megfile’s APIs, SmartPath, configuration, and CLI to perform filesystem tasks easy across local FS, S3/OSS-style object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.
---

# Megfile

## Overview

Guide to using megfile’s unified APIs, SmartPath, configuration, and CLI for file operations across various backends, like local FS, S3/OSS-compatible object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

## Quick Start
- Install base: `pip install megfile`; add extras per backend (`megfile[cli]`, `megfile[hdfs]`, `megfile[webdav]`).
- Configure credentials/endpoints (env vars > config files). S3 examples: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_ENDPOINT_URL` / `OSS_ENDPOINT` / `AWS_ENDPOINT_URL_S3`, `AWS_S3_ADDRESSING_STYLE`.
- Path format: `protocol://bucket/key` or alias (e.g., `tos://`); bare POSIX paths are treated as `file://`.
- Import functional APIs (`from megfile import smart_open, smart_sync, ...`) or SmartPath (`from megfile.smart_path import SmartPath`).

## Supported Protocols & Extras
- Local FS (`file://` or bare paths) — base install.
- S3/OSS-compatible (`s3://`, plus aliases) — base install.
- SFTP (`sftp://`) — install `megfile[cli]` or `megfile` with SFTP deps.
- HTTP/HTTPS (`http://`, `https://`) — base install.
- Stdio (`stdio://`) — base install.
- HDFS (`hdfs://`) — install `megfile[hdfs]`.
- WebDAV (`webdav://`) — install `megfile[webdav]`.
- Full Protocol path format reference: `references/path_format.md`.

## Core Tasks

### File IO
- `smart_open(path, mode='r', encoding=None, **options)`: open binary/text handles.
- Convenience loaders/savers: `smart_load_content` / `smart_save_content` (bytes), `smart_load_text` / `smart_save_text` (str), `smart_save_as(file_obj, path)`, `smart_load_from(path)` returns BinaryIO.
- `smart_combine_open(glob, mode='rb', open_func=smart_open)`: sequentially reads multiple files as one stream.

### Existence & Metadata
- `smart_exists`, `smart_isfile`, `smart_isdir`, `smart_islink`, `smart_isabs`.
- `smart_access(path, mode=Access.READ/WRITE)`.
- `smart_stat` / `smart_lstat` (via SmartPath), `smart_getsize`, `smart_getmtime`, `smart_getmd5(recalculate=False, followlinks=False)`.

### Listing & Globbing
- Directory traversal: `smart_listdir`, `smart_scandir`, `smart_walk`, `smart_scan`, `smart_scan_stat` (returns FileEntry/StatResult).
- Pattern matching: `smart_glob`, `smart_iglob`, `smart_glob_stat` for stat-rich globbing.

### Data Transfer & Lifecycle
- Copy/sync: `smart_copy`, `smart_sync`, `smart_sync_with_progress` (progress-friendly wrapper), `smart_concat` to merge multiple sources.
- Moves/deletes: `smart_move`, `smart_rename`, `smart_remove`, `smart_unlink`, `smart_touch`, `smart_makedirs`.
- Links: `smart_symlink`, `smart_readlink`.

### Path Utilities
- `smart_path_join`, `smart_abspath`, `smart_realpath`, `smart_relpath`, `smart_isabs`.
- SmartPath mirrors pathlib semantics but routes to the right backend: `path = SmartPath("s3://bucket/key"); path.exists(); path.open(mode="rb")`.

### Caching
- `smart_cache(path, cacher=SmartCacher, **options)`: cache remote resources locally for tools that only support local files. 

## Configuration
- Use CLI helpers to persist credentials/endpoints; environment variables take precedence.
- Profiles enable multiple endpoints (e.g., `s3+prod://...`). See `references/configuration/` for protocol-specific flags and env vars.
- Full config reference: `references/configuration/`.

## CLI Essentials
- Install CLI extras: `pip install 'megfile[cli]'`.
- Common commands (ls/cp/sync/stat/md5sum/mkdir/rm/touch) mirror POSIX semantics across backends.
- Completion scripts: `megfile completion zsh`.
- Full command list and flags: `references/cli.md`.

## Usage Notes
- Prefer smart_* for protocol-agnostic code paths; avoid branching per backend.
- Ensure required extras are installed for target protocols before invoking APIs.
- For high-volume sync/copy, supply `map_func` (e.g., `ThreadPoolExecutor.map`) and `callback` to report progress.
- Use aliases via `megfile config alias <alias> <protocol>` to shorten paths (e.g., `tos://`).

## References
- API surface: `references/megfile.smart.md` and `references/megfile.smart_path.md`.
- Configuration flags, env vars, and profiles: `references/configuration/`
- CLI commands and flags: `references/cli.md`
- Full Protocol path format reference: `references/path_format.md`.
- Glob patterns reference: `references/advanced/glob.md`.