File System

Don’t trust your memory — write it down.

Use this page when a demo, test, script, or extension needs predictable local files without hand-rolling path handling and serialization details. The File System utilities center on two ideas: pj(...) for portable paths and a serialization family for consistent read/write across JSON, YAML, text, JSONL, jstr, binary, and related formats.

1. Why a Unified File System Utility

Without a unified file system utility, every script and demo re-implements path joining, alias resolution, encoding handling, and serialization. The results are inconsistent: one module uses os.path.join(...) with hardcoded relative paths, another uses pathlib.Path with expanduser(), and a third hardcodes / separators that break on Windows. Serialization diverges the same way — one caller writes JSON with indent=2, another uses default=str, and a third hard-codes utf-8 encoding.

# Anti-pattern: every caller handles paths differently
import json
import os

demo_dir = os.path.join(os.path.expanduser("~"), "heavenbase-demo", "data")
config_path = "../config.yaml"  # breaks depending on working directory
cache_dir = "C:/Users/me/.cache/heavenbase/" if os.name == "nt" else "/home/me/.cache/heavenbase"

with open(os.path.join(demo_dir, "meta.json"), "w", encoding="utf-8") as handle:
    json.dump({"rows": 3}, handle, indent=2, default=str)

HeavenBase treats paths and serialization as paired infrastructure. pj(...) is the single path vocabulary: it expands ~, resolves environment variables, normalizes separators, optionally returns absolute paths, and resolves caller-provided aliases. The serialization family uses config-driven defaults (CM_HVNB at heavenbase.serialize.encoding and heavenbase.serialize.indent) so every caller gets consistent output without repeating those parameters.

from heavenbase.utils import dump_json, pj

# Portable paths and consistent serialization in one flow
project_root = pj(".", abs=True)
cache_file = pj("~/heavenbase-cache", "index.json", abs=True)
run_path = pj("%/runs/latest.json", aliases={"%": "demos/.temp"}, abs=True)

dump_json({"workspace": "docs-demo", "rows": 3}, run_path, sort_keys=True)

Both halves matter equally: you build a path with pj(...), then read or write through the matching serialization helper without re-specifying encoding, indentation, or platform details.

2. Two Core Ideas

File System utilities split cleanly into path vocabulary and serialization I/O. Everything else on this page — touching folders, listing files, copying artifacts, folder diagrams — supports those two flows.

2.1. `pj(...)` — Portable Paths

pj(...) is the canonical path join helper. It joins path parts, expands ~ and environment variables, normalizes separators, optionally returns an absolute path, and can resolve caller-provided aliases. Think of it as a small path vocabulary. . means the current working location, ~ expands to the user home, environment variables expand through the operating system, and aliases such as % or & can point to a package data folder or resource folder when the caller or config manager provides that alias map.

Utility pj(...) handles normal paths and caller aliases. CM_HVNB.pj(...) handles HeavenBase package aliases: %/ for package-local data and &/ for package resources.

2.2. Serialization Family — Config-Driven I/O

The serialization helpers wrap common read/write for the same reason configuration wraps hyperparameters: callers should not pass encoding, indentation, and platform path details through every function. Each format exposes the same naming pattern:

Prefix	Scope	Example
`loads_` / `dumps_`	In-memory strings or bytes	`dumps_json({"rows": 3})`
`load_` / `dump_`	File read/write	`dump_json(obj, path)`
`save_*`	Alias of `dump_*`	`save_txt("# Notes", path)`
`append_*`	Append one record or line	`append_jsonl(event, path)`

Text encodings default to heavenbase.serialize.encoding, which is utf-8 in the default config. JSON and YAML indentation default to heavenbase.serialize.indent, which is 4. Missing files return empty defaults ("", {}, [], or b"") unless you pass strict=True. This shows the benefit of the config-driven approach: you can change the default encoding or indentation without changing every caller. JSON helpers use HbJsonEncoder and HbJsonDecoder, which preserve common HeavenBase values such as datetimes, decimals, tuples, sets, and callable references — so artifacts round-trip without ad-hoc default=str handlers.

3. Build a Path

Use pj(...) for ordinary local paths:

from heavenbase.utils import pj

project_root = pj(".", abs=True)
cache_file = pj("~/heavenbase-cache", "index.json", abs=True)

Pass aliases when the first path segment is a project-specific shortcut:

from heavenbase.utils import pj

run_path = pj("%/runs/latest.json", aliases={"%": "demos/.temp"}, abs=True)
asset_path = pj("&/sample.txt", aliases={"&": "demos/assets"}, abs=True)

Use package aliases through the config manager:

from heavenbase.utils import CM_HVNB

config_db = CM_HVNB.pj("%/config.db", abs=True)
default_yaml = CM_HVNB.pj("&/configs/default.yaml", abs=True)

4. Prepare Files and Folders

Use touch_dir(...) and touch_file(...) before writing artifacts. Both create missing parents and return an absolute path.

from heavenbase.utils import exists_file, touch_dir, touch_file

root = touch_dir("demos/.temp/report")
summary = touch_file(f"{root}/summary.md", content="# Summary")

assert exists_file(summary)

Use clear=True only when the folder or file should start empty:

from heavenbase.utils import touch_dir

run_dir = touch_dir("demos/.temp/run", clear=True)

5. Read and Write Data

Serialization is the second pillar of this page. Pick a format, call the matching helper, and let CM_HVNB supply encoding and indentation defaults. Combine with pj(...) and touch_dir(...) for a complete artifact workflow.

from heavenbase.utils import append_jsonl, dump_json, load_json, pj, save_txt, touch_dir

root = touch_dir(pj("demos", ".temp", "docs-run"))
meta_path = pj(root, "meta.json")

dump_json({"workspace": "docs-demo", "rows": 3}, meta_path, sort_keys=True)
save_txt("# Run Notes", pj(root, "notes.md"))
append_jsonl({"event": "created", "path": meta_path}, pj(root, "events.jsonl"))

meta = load_json(meta_path)

5.1. JSON

Use JSON for structured machine-readable artifacts — configs, run metadata, workspace snapshots, and interchange with other tools.

from heavenbase.utils import dump_json, dumps_json, load_json, loads_json

payload = {"workspace": "docs-demo", "items": (1, 2)}  # tuples round-trip
text = dumps_json(payload, sort_keys=True, compact=True)
restored = loads_json(text)

dump_json(payload, "demos/.temp/docs-run/meta.json", sort_keys=True)
assert load_json("demos/.temp/docs-run/meta.json")["items"] == (1, 2)

Pass sort_keys=True when deterministic key order matters (for example, before hashing). Pass compact=True or indent=None for one-line output.

5.2. YAML

Use YAML for human-editable structured files such as config templates and small manifests.

from heavenbase.utils import dump_yaml, dumps_yaml, load_yaml

spec = {"model": "demo", "batch_size": 8}
dump_yaml(spec, "demos/.temp/docs-run/spec.yaml")
assert load_yaml("demos/.temp/docs-run/spec.yaml")["batch_size"] == 8

YAML shares the same indentation default as JSON through heavenbase.serialize.indent.

5.3. Text

Use text helpers for logs, Markdown notes, plain reports, and any human-readable line-oriented output.

from heavenbase.utils import append_txt, load_txt, save_txt

save_txt("# Run Notes\n", "demos/.temp/docs-run/notes.md")
append_txt("event: created", "demos/.temp/docs-run/log.txt")
notes = load_txt("demos/.temp/docs-run/notes.md")

load_txt returns "" for a missing file unless strict=True. append_txt adds a trailing newline after each write.

5.4. JSONL

Use JSONL when a workflow appends one event or record at a time — run logs, streaming exports, and incremental audit trails.

from heavenbase.utils import append_jsonl, dump_jsonl, iter_jsonl, load_jsonl

append_jsonl({"event": "created", "idx": 1}, "demos/.temp/docs-run/events.jsonl")
append_jsonl({"event": "updated", "idx": 2}, "demos/.temp/docs-run/events.jsonl")

rows = load_jsonl("demos/.temp/docs-run/events.jsonl")
for row in iter_jsonl("demos/.temp/docs-run/events.jsonl"):
    print(row["event"])

Prefer append_jsonl for incremental writes and dump_jsonl when you have the full iterable in memory. Use iter_jsonl to stream large files without loading every line at once.

5.5. jstr — Python-Facing Text

jstr means “JSON when the top-level value is a container, otherwise string.” Use dumps_jstr, loads_jstr, dump_jstr, load_jstr, and save_jstr at text boundaries such as Tool results, memstate rows, and MCP payloads where structured values should stay machine-readable while scalars and other values stay human-readable through str(...). Rules:

Top-level dict or list values (including nested containers) serialize as compact JSON through the same encoder as dumps_json.
All other values, including int, float, bool, None, tuples, sets, and custom objects, use str(obj).
loads_jstr parses text that looks like a JSON object or array; invalid JSON in that shape, and all other text, is returned unchanged as a string.

Toolkit and MCP boundaries use this format by default. Each Tool keeps serializer=None and resolves to the shared jstr_serializer Capsule at execution time, so toolkit.run_to_str(...) and FastMCP serving route successful tool outputs through dumps_jstr. Local Python callers can still use toolkit.run(...) for raw objects.

from heavenbase.utils import dumps_jstr, loads_jstr, save_jstr

assert dumps_jstr(42) == "42"
assert dumps_jstr({"rows": 3, "items": [{"id": 1}]}) == '{"rows": 3, "items": [{"id": 1}]}'
assert loads_jstr('{"rows": 3}') == {"rows": 3}
assert loads_jstr("plain note") == "plain note"

save_jstr({"event": "created"}, "demos/.temp/docs-run/events.jstr")

5.6. Binary

Use binary helpers for raw byte payloads — model weights, compressed blobs, image bytes, and other non-text artifacts.

from heavenbase.utils import dump_bin, load_bin, save_bin

payload = b"\x00\x01\x02"
save_bin(payload, "demos/.temp/docs-run/blob.bin")
assert load_bin("demos/.temp/docs-run/blob.bin") == payload

load_bin returns b"" for a missing file unless strict=True.

5.7. Hex and Base64

Use hex and Base64 helpers when byte payloads need a text-safe representation in files or logs.

from heavenbase.utils import dump_b64, dump_hex, load_b64, load_hex, dumps_b64, loads_b64

raw = b"hello"
assert loads_b64(dumps_b64(raw)) == raw

dump_hex("48656c6c6f", "demos/.temp/docs-run/hello.hex")
assert load_hex("demos/.temp/docs-run/hello.hex") == "48656c6c6f"

dump_b64(dumps_b64(raw), "demos/.temp/docs-run/hello.bin")
assert load_b64("demos/.temp/docs-run/hello.bin") == dumps_b64(raw)

dump_hex and dump_b64 decode a hex or Base64 string and write the resulting bytes. load_hex and load_b64 read a binary file and return a hex or Base64 text representation. Use dumps_b64 / loads_b64 for in-memory encoding without touching the filesystem.

5.8. Pickle

Use pickle only for trusted local artifacts produced by your own environment. Pickle can execute arbitrary code during load and must never cross a trust boundary.

from heavenbase.utils import dump_pkl, load_pkl

state = {"cache": {"rows": 3}}
dump_pkl(state, "demos/.temp/docs-run/state.pkl")
assert load_pkl("demos/.temp/docs-run/state.pkl") == state

load_pkl(...) reads Python pickle data. Use pickle only for trusted local artifacts produced by your own environment.

6. List, Copy, and Delete Artifacts

list_files(...) reads direct children. enum_files(...) walks recursively. Both return stable, case-insensitive order.

from heavenbase.utils import copy_path, delete_path, enum_files, list_files

for name in list_files("demos/.temp/docs-run", ext="json"):
    print(name)

for path in enum_files("demos/.temp/docs-run", ext=["json", "jsonl"], abs=True):
    print(path)

copy_path("demos/.temp/docs-run", "demos/.temp/docs-run-copy", mode="merge")
delete_path("demos/.temp/docs-run-copy")

Copy helpers use explicit conflict modes. Files support replace, skip, and strict; directories also support merge.

7. Show a Folder Diagram

Use folder_diagram(...) for compact diagnostics in logs, reports, and docs snippets.

from heavenbase.utils import folder_diagram

print(folder_diagram("demos/.temp/docs-run", annotations={"meta.json": "run metadata"}, limit=12))

delete_dir(...) and delete_path(...) remove local data. Keep demo cleanup targets under known temporary folders such as demos/.temp/.

Further Exploration

Related resources:

Configuration - config-driven defaults used by serialization helpers.
Hash - deterministic file-safe identifiers and fingerprints.
Miscs - command execution around local artifacts.

Introduction

Quickstart

Core Concepts

Developer

Utilities Toolbox

LLM

Agent Runtime

Community

1. Why a Unified File System Utility

2. Two Core Ideas

2.1. `pj(...)` — Portable Paths

2.2. Serialization Family — Config-Driven I/O

3. Build a Path

4. Prepare Files and Folders

5. Read and Write Data

5.1. JSON

5.2. YAML

5.3. Text

5.4. JSONL

5.5. jstr — Python-Facing Text

5.6. Binary

5.7. Hex and Base64

5.8. Pickle

6. List, Copy, and Delete Artifacts

7. Show a Folder Diagram

Further Exploration

​1. Why a Unified File System Utility

​2. Two Core Ideas

​2.1. pj(...) — Portable Paths

​2.2. Serialization Family — Config-Driven I/O

​3. Build a Path

​4. Prepare Files and Folders

​5. Read and Write Data

​5.1. JSON

​5.2. YAML

​5.3. Text

​5.4. JSONL

​5.5. jstr — Python-Facing Text

​5.6. Binary

​5.7. Hex and Base64

​5.8. Pickle

​6. List, Copy, and Delete Artifacts

​7. Show a Folder Diagram

​Further Exploration

1. Why a Unified File System Utility

2. Two Core Ideas

2.1. `pj(...)` — Portable Paths

2.2. Serialization Family — Config-Driven I/O

3. Build a Path

4. Prepare Files and Folders

5. Read and Write Data

5.1. JSON

5.2. YAML

5.3. Text

5.4. JSONL

5.5. jstr — Python-Facing Text

5.6. Binary

5.7. Hex and Base64

5.8. Pickle

6. List, Copy, and Delete Artifacts

7. Show a Folder Diagram

Further Exploration