Mastering Node.js File System Operations: Copy, Move, Rename, Delete, and ListA deep dive into Node.js core file system APIs for building reliable file manipulation tools

Introduction

File system operations form the backbone of countless software applications—from build tools and deployment scripts to data processing pipelines and content management systems. Node.js provides comprehensive file system capabilities through its built-in fs module, offering both low-level primitives and higher-level abstractions for manipulating files and directories. Unlike browser JavaScript, which runs in a sandboxed environment with limited file access, Node.js grants full access to the underlying operating system's file system, making it a powerful platform for system utilities, automation tools, and server-side applications that need to read, write, and organize files.

Understanding Node.js file system operations goes beyond simply knowing which function to call. Professional file manipulation requires handling edge cases like race conditions, permission errors, and cross-platform path differences. It demands knowledge of when to use synchronous versus asynchronous operations, how to ensure atomicity of multi-step operations, and how to handle errors gracefully without leaving the file system in inconsistent states. This article explores the five fundamental file system operations—listing, copying, moving, renaming, and deleting—with a focus on production-ready patterns, error handling strategies, and performance considerations that separate robust implementations from fragile ones.

Node.js File System API Architecture

The Node.js fs module has evolved through three distinct API styles, each serving different use cases and programming paradigms. The original callback-based API (fs.readFile(path, callback)) follows Node.js's traditional error-first callback convention, where the first callback parameter contains an error object if the operation failed, or null if it succeeded. While this API remains fully supported for backward compatibility, its nested callback structure becomes unwieldy for operations involving multiple steps, leading to the infamous "callback pyramid" that complicates error handling and control flow.

Synchronous operations (fs.readFileSync(path)) provide a simpler programming model by blocking execution until the operation completes, either returning a result or throwing an exception. These functions are invaluable for initialization code, configuration loading, and command-line tools where blocking behavior is acceptable or even desirable. However, using synchronous file operations in server request handlers or event loop-dependent code creates performance bottlenecks, preventing Node.js from handling other work while waiting for disk I/O. The synchronous API is not inherently bad—it's a tool that excels in specific contexts but can severely harm application performance when misapplied.

The promises-based API, introduced in Node.js 10 and accessed via fs.promises or require('fs').promises, represents the modern approach to asynchronous file operations. These functions return Promises that resolve with operation results or reject with errors, enabling clean asynchronous code using async/await syntax. This API eliminates callback nesting while maintaining non-blocking behavior, making it the preferred choice for most contemporary Node.js applications. Importantly, all three API styles—callbacks, synchronous, and promises—provide the same underlying functionality; they differ only in how they handle asynchronicity and errors. Choosing appropriately among them based on context demonstrates engineering maturity.

Listing Directory Contents

Reading directory contents represents the foundation of many file system operations—from build tools discovering source files to backup utilities cataloging data. The fs.readdir() function (and its variants) returns an array of filenames within a specified directory, but the basic operation lacks critical information like file types, sizes, and permissions. Node.js provides two approaches for obtaining richer directory information: calling fs.stat() on each entry after reading the directory, or using the withFileTypes option that returns Dirent objects containing type information without additional system calls.

The Dirent (directory entry) API significantly improves performance for operations that need to distinguish files from directories. Each Dirent object includes methods like isFile(), isDirectory(), isSymbolicLink(), and isSocket() that report entry types without requiring separate stat() calls. This matters because stat operations involve additional disk I/O—for a directory containing thousands of entries, eliminating these extra system calls can reduce operation time by an order of magnitude. When you need only filenames or types, readdir() with withFileTypes provides optimal performance. When you need detailed metadata like size or modification time, the separate stat() calls become necessary.

Recursive directory traversal requires careful implementation to handle deep hierarchies, symbolic links, and permission errors gracefully. A naive recursive approach reads a directory, processes files, and recursively calls itself on subdirectories—simple but vulnerable to stack overflow on extremely deep trees and infinite loops from circular symbolic links. Production implementations should use iterative algorithms with explicit stacks or queues, track visited inodes to detect cycles, and handle permission denied errors without failing the entire traversal. The pattern of separating traversal logic (what to visit) from action logic (what to do with each entry) enables reusable traversal utilities.

const fs = require('fs').promises;
const path = require('path');

/**
 * List directory contents with metadata using Dirent API
 * More efficient than separate stat calls
 */
async function listDirectory(dirPath, options = {}) {
  try {
    const entries = await fs.readdir(dirPath, { withFileTypes: true });
    
    const results = entries.map(entry => ({
      name: entry.name,
      path: path.join(dirPath, entry.name),
      isFile: entry.isFile(),
      isDirectory: entry.isDirectory(),
      isSymlink: entry.isSymbolicLink()
    }));

    // Apply filters if provided
    if (options.filesOnly) {
      return results.filter(r => r.isFile);
    }
    if (options.directoriesOnly) {
      return results.filter(r => r.isDirectory);
    }

    return results;

  } catch (error) {
    if (error.code === 'ENOENT') {
      throw new Error(`Directory not found: ${dirPath}`);
    }
    if (error.code === 'EACCES') {
      throw new Error(`Permission denied: ${dirPath}`);
    }
    throw error;
  }
}

/**
 * Recursively traverse directory tree
 * Iterative implementation to avoid stack overflow
 */
async function traverseDirectory(rootPath, callback, options = {}) {
  const maxDepth = options.maxDepth || Infinity;
  const followSymlinks = options.followSymlinks || false;
  const visitedInodes = new Set();
  
  // Queue entries: { path, depth }
  const queue = [{ path: rootPath, depth: 0 }];
  const results = [];

  while (queue.length > 0) {
    const { path: currentPath, depth } = queue.shift();

    // Check depth limit
    if (depth > maxDepth) {
      continue;
    }

    try {
      // Get stats to detect cycles (via inode)
      const stats = await fs.stat(currentPath);
      
      // Detect circular references (Unix-specific)
      if (stats.ino) {
        const inodeKey = `${stats.dev}:${stats.ino}`;
        if (visitedInodes.has(inodeKey)) {
          console.warn(`Skipping circular reference: ${currentPath}`);
          continue;
        }
        visitedInodes.add(inodeKey);
      }

      // Process current entry with callback
      const continueTraversal = await callback({
        path: currentPath,
        stats,
        depth
      });

      // If callback returns false, skip this branch
      if (continueTraversal === false) {
        continue;
      }

      // If directory, add children to queue
      if (stats.isDirectory()) {
        const entries = await fs.readdir(currentPath, { withFileTypes: true });
        
        for (const entry of entries) {
          const entryPath = path.join(currentPath, entry.name);
          
          // Handle symlinks based on options
          if (entry.isSymbolicLink() && !followSymlinks) {
            continue;
          }

          queue.push({ path: entryPath, depth: depth + 1 });
        }
      }

    } catch (error) {
      // Handle permission errors gracefully
      if (error.code === 'EACCES' || error.code === 'EPERM') {
        if (options.onError) {
          options.onError(error, currentPath);
        } else {
          console.warn(`Permission denied: ${currentPath}`);
        }
        continue;
      }
      
      // Rethrow unexpected errors
      throw error;
    }
  }

  return results;
}

/**
 * Example: Find all files matching a pattern recursively
 */
async function findFiles(rootPath, pattern, options = {}) {
  const regex = new RegExp(pattern);
  const matches = [];

  await traverseDirectory(rootPath, async (entry) => {
    if (entry.stats.isFile() && regex.test(entry.path)) {
      matches.push({
        path: entry.path,
        size: entry.stats.size,
        modified: entry.stats.mtime
      });
    }
    return true; // Continue traversal
  }, options);

  return matches;
}

module.exports = { listDirectory, traverseDirectory, findFiles };

Copying Files and Directories

File copying in Node.js has evolved significantly over the module's history. Early versions required manually reading source files and writing to destinations, but modern Node.js provides fs.copyFile() for individual files and fs.cp() (added in Node.js 16.7.0) for recursive directory copying. The fs.copyFile() function operates at the system call level, often leveraging efficient kernel-space operations like copy-on-write when available, making it substantially faster than userspace read-write loops. For copying large files, this performance difference becomes dramatic—system-level copying can be 10-50 times faster than manual streaming, depending on the underlying file system and operating system.

The fs.copyFile() function accepts a mode parameter that controls copying behavior through bitwise flags. The fs.constants.COPYFILE_EXCL flag causes the operation to fail if the destination exists, preventing accidental overwrites—critical for operations where data loss would be catastrophic. The COPYFILE_FICLONE flag attempts to use copy-on-write cloning when supported by the file system (like Btrfs or APFS), creating essentially instant copies of large files that share disk blocks until modified. The COPYFILE_FICLONE_FORCE flag makes cloning required, failing the operation if the file system doesn't support it. These flags enable precise control over copying semantics, allowing applications to enforce safety constraints or optimize for performance.

Copying directories recursively involves more complexity than single files because it requires replicating entire hierarchies while preserving permissions, timestamps, and handling symbolic links appropriately. The fs.cp() function added in recent Node.js versions handles this complexity, but understanding manual implementation illuminates the challenges. A robust recursive copy must create destination directories before copying their contents, handle symbolic links based on policy (copy the link itself versus copy the target), preserve file metadata when desired, and handle errors gracefully without leaving partially copied hierarchies. The traversal algorithm from the previous section provides the foundation—augment it with copy operations at each node.

const fs = require('fs').promises;
const path = require('path');

/**
 * Copy a single file with options for overwrite control
 */
async function copyFile(sourcePath, destPath, options = {}) {
  try {
    let mode = 0;

    // Set copy mode flags
    if (options.errorIfExists) {
      mode |= fs.constants.COPYFILE_EXCL;
    }
    if (options.useCloneIfAvailable) {
      mode |= fs.constants.COPYFILE_FICLONE;
    }

    await fs.copyFile(sourcePath, destPath, mode);

    // Preserve timestamps if requested
    if (options.preserveTimestamps) {
      const stats = await fs.stat(sourcePath);
      await fs.utimes(destPath, stats.atime, stats.mtime);
    }

    return { success: true, source: sourcePath, destination: destPath };

  } catch (error) {
    if (error.code === 'EEXIST') {
      throw new Error(`Destination already exists: ${destPath}`);
    }
    if (error.code === 'ENOENT') {
      throw new Error(`Source file not found: ${sourcePath}`);
    }
    if (error.code === 'EACCES' || error.code === 'EPERM') {
      throw new Error(`Permission denied: ${error.message}`);
    }
    throw error;
  }
}

/**
 * Recursively copy directory (manual implementation for understanding)
 * For production, use fs.cp() in Node.js 16.7+
 */
async function copyDirectory(sourceDir, destDir, options = {}) {
  const { 
    overwrite = false,
    preserveTimestamps = false,
    filter = null,
    onProgress = null
  } = options;

  const stats = await fs.stat(sourceDir);
  
  if (!stats.isDirectory()) {
    throw new Error(`Source is not a directory: ${sourceDir}`);
  }

  // Create destination directory
  try {
    await fs.mkdir(destDir, { recursive: true });
  } catch (error) {
    if (error.code !== 'EEXIST') {
      throw error;
    }
  }

  // Read source directory
  const entries = await fs.readdir(sourceDir, { withFileTypes: true });
  let copiedCount = 0;
  let skippedCount = 0;

  for (const entry of entries) {
    const sourcePath = path.join(sourceDir, entry.name);
    const destPath = path.join(destDir, entry.name);

    // Apply filter if provided
    if (filter && !filter(sourcePath, entry)) {
      skippedCount++;
      continue;
    }

    try {
      if (entry.isDirectory()) {
        // Recursive copy for subdirectories
        await copyDirectory(sourcePath, destPath, options);
      } else if (entry.isFile()) {
        // Check if destination exists
        if (!overwrite) {
          try {
            await fs.access(destPath);
            // File exists and overwrite is false
            skippedCount++;
            continue;
          } catch {
            // File doesn't exist, proceed with copy
          }
        }

        // Copy the file
        await fs.copyFile(sourcePath, destPath);

        if (preserveTimestamps) {
          const sourceStats = await fs.stat(sourcePath);
          await fs.utimes(destPath, sourceStats.atime, sourceStats.mtime);
        }

        copiedCount++;

      } else if (entry.isSymbolicLink()) {
        // Copy symlink itself, not its target
        const linkTarget = await fs.readlink(sourcePath);
        await fs.symlink(linkTarget, destPath);
        copiedCount++;
      }

      if (onProgress) {
        onProgress({ copied: copiedCount, skipped: skippedCount });
      }

    } catch (error) {
      if (options.continueOnError) {
        console.error(`Error copying ${sourcePath}: ${error.message}`);
        skippedCount++;
      } else {
        throw error;
      }
    }
  }

  return { copiedCount, skippedCount };
}

/**
 * Modern approach using fs.cp() (Node.js 16.7+)
 */
async function copyDirectoryModern(sourceDir, destDir, options = {}) {
  try {
    await fs.cp(sourceDir, destDir, {
      recursive: true,
      errorOnExist: !options.overwrite,
      force: options.overwrite,
      preserveTimestamps: options.preserveTimestamps || false,
      filter: options.filter,
      verbatimSymlinks: true // Copy symlinks as-is
    });

    return { success: true };

  } catch (error) {
    if (error.code === 'ERR_FS_CP_DIR_TO_NON_DIR') {
      throw new Error('Cannot copy directory to a file');
    }
    throw error;
  }
}

module.exports = { copyFile, copyDirectory, copyDirectoryModern };

Moving and Renaming Files

Moving and renaming files in Node.js uses the same underlying function—fs.rename()—because at the file system level, these operations are identical: they change the path associated with an inode (the data structure representing a file). The fs.rename() function attempts an atomic rename operation when source and destination reside on the same file system, making it extremely efficient and safe. However, when moving files across different file systems or partitions, the operation cannot be atomic at the kernel level, and Node.js must fall back to a copy-then-delete sequence, which introduces failure modes that pure renames avoid.

Cross-device moves present the most significant challenge for reliable file moving operations. When fs.rename() fails with an EXDEV error (cross-device link not permitted), the application must implement the move manually: copy the source to the destination, verify the copy succeeded, then delete the source. This multi-step process isn't atomic—failures between copy and delete leave files duplicated, while failures during copy might leave partial destination files. Proper implementation must handle these edge cases: verify destination integrity, use temporary destination names to avoid clobbering, and implement rollback logic for partial failures.

Renaming also serves as the foundation for atomic file updates—a critical pattern for ensuring configuration files, databases, or other critical data never appear partially written. The strategy writes new content to a temporary file, ensuring all data reaches disk via fsync(), then atomically renames the temporary file over the target. Because rename operations on the same file system are atomic at the kernel level, observers never see partial content. This pattern, sometimes called "write-rename" or "atomic replace," is fundamental to building reliable systems and appears in everything from text editors to database engines.

const fs = require('fs').promises;
const path = require('path');

/**
 * Rename/move a file with cross-device support
 */
async function moveFile(sourcePath, destPath, options = {}) {
  const { overwrite = false } = options;

  try {
    // Check if destination exists
    if (!overwrite) {
      try {
        await fs.access(destPath);
        throw new Error(`Destination already exists: ${destPath}`);
      } catch (error) {
        if (error.code !== 'ENOENT') {
          throw error;
        }
        // ENOENT is expected - destination doesn't exist
      }
    }

    // Attempt atomic rename
    await fs.rename(sourcePath, destPath);
    return { method: 'rename', success: true };

  } catch (error) {
    // Handle cross-device move
    if (error.code === 'EXDEV') {
      return await copyAndDelete(sourcePath, destPath, options);
    }

    // Handle other errors
    if (error.code === 'ENOENT') {
      throw new Error(`Source file not found: ${sourcePath}`);
    }
    if (error.code === 'EACCES' || error.code === 'EPERM') {
      throw new Error(`Permission denied: ${error.message}`);
    }
    
    throw error;
  }
}

/**
 * Copy then delete for cross-device moves
 */
async function copyAndDelete(sourcePath, destPath, options = {}) {
  let tempDestPath = null;

  try {
    // Use temporary destination to avoid partial overwrites
    const destDir = path.dirname(destPath);
    const destName = path.basename(destPath);
    tempDestPath = path.join(destDir, `.${destName}.${Date.now()}.tmp`);

    // Copy to temporary location
    await fs.copyFile(sourcePath, tempDestPath);

    // Verify copy if paranoid mode enabled
    if (options.verify) {
      const [sourceStats, destStats] = await Promise.all([
        fs.stat(sourcePath),
        fs.stat(tempDestPath)
      ]);

      if (sourceStats.size !== destStats.size) {
        throw new Error('Copy verification failed: size mismatch');
      }
    }

    // Atomic rename temp to final destination
    await fs.rename(tempDestPath, destPath);

    // Delete source file
    await fs.unlink(sourcePath);

    return { method: 'copy-delete', success: true };

  } catch (error) {
    // Cleanup temporary file if it exists
    if (tempDestPath) {
      try {
        await fs.unlink(tempDestPath);
      } catch (cleanupError) {
        console.warn(`Failed to cleanup temp file: ${tempDestPath}`);
      }
    }
    throw error;
  }
}

/**
 * Atomic file update using write-rename pattern
 * Ensures readers never see partial content
 */
async function atomicWriteFile(filePath, content, options = {}) {
  const dir = path.dirname(filePath);
  const filename = path.basename(filePath);
  const tempPath = path.join(dir, `.${filename}.${process.pid}.${Date.now()}.tmp`);

  try {
    // Write to temporary file
    await fs.writeFile(tempPath, content, options.encoding || 'utf-8');

    // Ensure data is written to disk (not just buffered)
    if (options.fsync) {
      const fileHandle = await fs.open(tempPath, 'r+');
      try {
        await fileHandle.sync();
      } finally {
        await fileHandle.close();
      }
    }

    // Preserve original file permissions if it exists
    if (options.preserveMode) {
      try {
        const stats = await fs.stat(filePath);
        await fs.chmod(tempPath, stats.mode);
      } catch (error) {
        if (error.code !== 'ENOENT') {
          throw error;
        }
        // Original file doesn't exist, use default mode
      }
    }

    // Atomic rename
    await fs.rename(tempPath, filePath);

    return { success: true, path: filePath };

  } catch (error) {
    // Cleanup temp file on error
    try {
      await fs.unlink(tempPath);
    } catch {
      // Ignore cleanup errors
    }
    throw error;
  }
}

/**
 * Batch rename files using a transformation function
 */
async function batchRename(dirPath, transformFn, options = {}) {
  const { dryRun = false } = options;
  const entries = await fs.readdir(dirPath, { withFileTypes: true });
  const operations = [];

  for (const entry of entries) {
    if (!entry.isFile() && !options.includeDirectories) {
      continue;
    }

    const oldName = entry.name;
    const newName = transformFn(oldName);

    if (newName && newName !== oldName) {
      const sourcePath = path.join(dirPath, oldName);
      const destPath = path.join(dirPath, newName);
      
      operations.push({ source: sourcePath, dest: destPath, oldName, newName });
    }
  }

  // Check for conflicts
  const destNames = new Set();
  for (const op of operations) {
    if (destNames.has(op.newName)) {
      throw new Error(`Rename conflict: multiple files would be renamed to '${op.newName}'`);
    }
    destNames.add(op.newName);
  }

  // Execute renames
  if (!dryRun) {
    for (const op of operations) {
      await fs.rename(op.source, op.dest);
    }
  }

  return operations;
}

module.exports = { moveFile, atomicWriteFile, batchRename };

Deleting Files and Directories

File deletion appears deceptively simple but contains numerous subtleties that affect reliability and safety. The fs.unlink() function removes files by decrementing their link count—when a file's link count reaches zero and no processes hold it open, the operating system reclaims its disk space. This behavior means unlink() succeeds even if other processes currently access the file, and those processes continue operating on the now-deleted file until they close it. This Unix semantics differs from Windows, where attempting to delete an open file typically fails with a sharing violation error, creating cross-platform inconsistencies that applications must handle.

Directory deletion requires the directory to be empty, enforced by the operating system to prevent accidental data loss. The fs.rmdir() function removes empty directories but fails if contents remain. For recursive deletion, Node.js added fs.rm() with a recursive: true option (Node.js 14.14.0) that removes directories and all their contents. This operation is inherently dangerous—it can delete thousands of files in milliseconds without undo capability. Production implementations should include safety checks: verify the path doesn't target system directories, require explicit confirmation for large deletions, implement dry-run modes, and log deleted paths for forensic purposes. Some applications create trash/recycling directories instead of true deletion, moving files to recoverable locations as a safety net.

Handling deletion errors properly requires understanding the distinction between expected failures (file doesn't exist) and unexpected failures (permission denied, disk full). When deleting temporary files or cleaning up, treating ENOENT (file not found) as success rather than failure simplifies logic—if the goal is ensuring a file doesn't exist, discovering it already doesn't exist accomplishes that goal. However, when deleting user-specified files, ENOENT likely indicates an error in the path or a race condition, and should be reported. This context-dependent error handling reflects mature engineering judgment about what constitutes success for different operations.

const fs = require('fs').promises;
const path = require('path');

/**
 * Safely delete a file with proper error handling
 */
async function deleteFile(filePath, options = {}) {
  const { ignoreIfMissing = false } = options;

  try {
    await fs.unlink(filePath);
    return { success: true, path: filePath };

  } catch (error) {
    if (error.code === 'ENOENT') {
      if (ignoreIfMissing) {
        return { success: true, path: filePath, alreadyDeleted: true };
      }
      throw new Error(`File not found: ${filePath}`);
    }

    if (error.code === 'EISDIR') {
      throw new Error(`Path is a directory, not a file: ${filePath}`);
    }

    if (error.code === 'EACCES' || error.code === 'EPERM') {
      throw new Error(`Permission denied: ${filePath}`);
    }

    throw error;
  }
}

/**
 * Recursively delete directory with safety checks
 */
async function deleteDirectory(dirPath, options = {}) {
  const {
    recursive = true,
    safetyCheck = true,
    dryRun = false,
    onProgress = null
  } = options;

  // Safety checks to prevent catastrophic deletions
  if (safetyCheck) {
    const resolvedPath = path.resolve(dirPath);
    const dangerousPaths = [
      '/',
      '/home',
      '/root',
      '/etc',
      '/usr',
      '/var',
      path.resolve(process.env.HOME || '/'),
      path.resolve(process.cwd())
    ];

    if (dangerousPaths.includes(resolvedPath)) {
      throw new Error(`Refusing to delete protected directory: ${resolvedPath}`);
    }

    // Check path length as heuristic for accidental root deletion
    const depth = resolvedPath.split(path.sep).filter(Boolean).length;
    if (depth < 3) {
      throw new Error(`Path too short, possibly dangerous: ${resolvedPath}`);
    }
  }

  // Verify directory exists
  let stats;
  try {
    stats = await fs.stat(dirPath);
  } catch (error) {
    if (error.code === 'ENOENT') {
      return { success: true, alreadyDeleted: true };
    }
    throw error;
  }

  if (!stats.isDirectory()) {
    throw new Error(`Not a directory: ${dirPath}`);
  }

  if (dryRun) {
    // Calculate what would be deleted
    const items = await collectAllItems(dirPath);
    return { dryRun: true, wouldDelete: items };
  }

  // Modern approach for Node.js 14.14+
  if (recursive && fs.rm) {
    await fs.rm(dirPath, { 
      recursive: true, 
      force: options.force || false,
      maxRetries: options.maxRetries || 3,
      retryDelay: options.retryDelay || 100
    });
    return { success: true, path: dirPath };
  }

  // Fallback: manual recursive deletion
  return await deleteDirectoryRecursive(dirPath, options);
}

/**
 * Manual recursive deletion implementation
 */
async function deleteDirectoryRecursive(dirPath, options = {}) {
  const entries = await fs.readdir(dirPath, { withFileTypes: true });
  let deletedCount = 0;

  // Delete all contents first
  for (const entry of entries) {
    const entryPath = path.join(dirPath, entry.name);

    try {
      if (entry.isDirectory()) {
        await deleteDirectoryRecursive(entryPath, options);
      } else {
        await fs.unlink(entryPath);
      }
      deletedCount++;

      if (options.onProgress) {
        options.onProgress({ deleted: deletedCount, current: entryPath });
      }

    } catch (error) {
      if (options.continueOnError) {
        console.error(`Failed to delete ${entryPath}: ${error.message}`);
      } else {
        throw error;
      }
    }
  }

  // Now delete the empty directory itself
  await fs.rmdir(dirPath);
  deletedCount++;

  return { deletedCount, path: dirPath };
}

/**
 * Collect all items in directory for dry-run reporting
 */
async function collectAllItems(dirPath) {
  const items = [];
  const entries = await fs.readdir(dirPath, { withFileTypes: true });

  for (const entry of entries) {
    const entryPath = path.join(dirPath, entry.name);
    items.push(entryPath);

    if (entry.isDirectory()) {
      const subitems = await collectAllItems(entryPath);
      items.push(...subitems);
    }
  }

  return items;
}

/**
 * Safe cleanup of temporary files pattern
 */
async function cleanupTemp(tempDir, options = {}) {
  const { maxAge = 24 * 60 * 60 * 1000 } = options; // 24 hours default
  const now = Date.now();
  const entries = await fs.readdir(tempDir, { withFileTypes: true });
  let cleaned = 0;

  for (const entry of entries) {
    const entryPath = path.join(tempDir, entry.name);

    try {
      const stats = await fs.stat(entryPath);
      const age = now - stats.mtimeMs;

      if (age > maxAge) {
        if (entry.isDirectory()) {
          await fs.rm(entryPath, { recursive: true, force: true });
        } else {
          await fs.unlink(entryPath);
        }
        cleaned++;
      }
    } catch (error) {
      // Continue cleaning even if individual items fail
      console.warn(`Failed to clean ${entryPath}: ${error.message}`);
    }
  }

  return { cleaned, path: tempDir };
}

module.exports = { deleteFile, deleteDirectory, cleanupTemp, moveFile };

Error Handling and Cross-Platform Considerations

Robust file system code must handle a wide variety of error conditions, each requiring appropriate recovery strategies. Permission errors (EACCES, EPERM) indicate the process lacks necessary rights to perform the operation—attempting to retry won't help unless the application can escalate privileges or prompt for user action. "No such file or directory" errors (ENOENT) might indicate typos, race conditions where another process deleted the file, or simply that expected files haven't been created yet. "File exists" errors (EEXIST) occur when creating files or directories that already exist, often requiring the application to decide whether to overwrite, skip, or fail. Understanding error code semantics enables appropriate handling rather than treating all errors identically.

Cross-platform path handling represents one of the most common sources of bugs in Node.js file system code. Windows uses backslashes as path separators and supports drive letters, while Unix systems use forward slashes and have a single root. The path module abstracts these differences, but only when used consistently—hardcoded slashes or string concatenation for paths creates platform-specific bugs. Always use path.join(), path.resolve(), and path.dirname() for path manipulation, never string concatenation. Additionally, Windows has reserved filenames (CON, PRN, AUX, NUL, COM1-COM9, LPT1-LPT9) that cause errors when used, and different maximum path length constraints (traditionally 260 characters, though modern Windows can be configured for longer paths).

File system race conditions occur when the state checked by one operation changes before a subsequent operation completes. The classic time-of-check-time-of-use (TOCTOU) bug involves checking whether a file exists, then acting based on that check—but another process might create or delete the file between check and action. Instead of checking then acting, use atomic operations that fail gracefully: try to open the file with appropriate flags (like O_CREAT | O_EXCL to fail if it exists), or use fs.copyFile() with COPYFILE_EXCL to prevent overwrites atomically. When atomic operations aren't available, accept that race conditions exist, implement retry logic with exponential backoff, and design systems to be resilient to occasional failures rather than assuming operations always succeed.

const fs = require('fs').promises;
const path = require('path');

/**
 * Error code classification and handling strategies
 */
const ERROR_STRATEGIES = {
  // Temporary errors - retry may help
  retriable: ['EAGAIN', 'EBUSY', 'EMFILE', 'ENFILE'],
  
  // Permission errors - retry won't help without privilege change
  permission: ['EACCES', 'EPERM'],
  
  // Path errors - indicates programming error or race condition
  path: ['ENOENT', 'ENOTDIR', 'EISDIR'],
  
  // Resource errors - system limits reached
  resource: ['ENOSPC', 'EDQUOT', 'ENOMEM'],
  
  // Cross-device errors - require different approach
  crossDevice: ['EXDEV']
};

/**
 * Classify error for appropriate handling
 */
function classifyError(error) {
  for (const [category, codes] of Object.entries(ERROR_STRATEGIES)) {
    if (codes.includes(error.code)) {
      return category;
    }
  }
  return 'unknown';
}

/**
 * Retry operation with exponential backoff for retriable errors
 */
async function retryOperation(operation, options = {}) {
  const {
    maxRetries = 3,
    initialDelay = 100,
    maxDelay = 5000,
    backoffFactor = 2
  } = options;

  let lastError;
  let delay = initialDelay;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      lastError = error;
      const errorType = classifyError(error);

      // Only retry for retriable errors
      if (errorType !== 'retriable' || attempt === maxRetries) {
        throw error;
      }

      // Wait before retrying
      await new Promise(resolve => setTimeout(resolve, delay));
      delay = Math.min(delay * backoffFactor, maxDelay);
    }
  }

  throw lastError;
}

/**
 * Normalize paths for cross-platform compatibility
 */
function normalizePath(inputPath) {
  // Resolve to absolute path
  const resolved = path.resolve(inputPath);
  
  // Normalize separators
  const normalized = path.normalize(resolved);
  
  return normalized;
}

/**
 * Validate filename against platform restrictions
 */
function validateFilename(filename) {
  const errors = [];

  // Windows reserved names
  const reserved = /^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(\.|$)/i;
  if (process.platform === 'win32' && reserved.test(filename)) {
    errors.push(`Reserved filename on Windows: ${filename}`);
  }

  // Invalid characters (Windows)
  if (process.platform === 'win32') {
    const invalidChars = /[<>:"|?*\x00-\x1F]/;
    if (invalidChars.test(filename)) {
      errors.push('Filename contains invalid characters');
    }
  }

  // Control characters (all platforms)
  if (/[\x00-\x1F]/.test(filename)) {
    errors.push('Filename contains control characters');
  }

  // Length validation
  if (filename.length > 255) {
    errors.push('Filename exceeds maximum length (255 characters)');
  }

  // Empty or just dots
  if (!filename || filename === '.' || filename === '..') {
    errors.push('Invalid filename');
  }

  return errors;
}

/**
 * Safe file operation wrapper with comprehensive error handling
 */
async function safeFileOperation(operation, operationName, options = {}) {
  const { allowRetry = false } = options;

  try {
    if (allowRetry) {
      return await retryOperation(operation, options);
    } else {
      return await operation();
    }

  } catch (error) {
    const errorType = classifyError(error);
    const enhancedError = new Error(
      `${operationName} failed: ${error.message} (${error.code || 'UNKNOWN'})`
    );
    
    enhancedError.code = error.code;
    enhancedError.originalError = error;
    enhancedError.errorType = errorType;
    
    // Add helpful context for common errors
    switch (errorType) {
      case 'permission':
        enhancedError.suggestion = 'Check file permissions and process user privileges';
        break;
      case 'resource':
        enhancedError.suggestion = 'Check available disk space or system resource limits';
        break;
      case 'crossDevice':
        enhancedError.suggestion = 'Source and destination are on different file systems';
        break;
      case 'path':
        enhancedError.suggestion = 'Verify path exists and is the expected type (file/directory)';
        break;
    }

    throw enhancedError;
  }
}

module.exports = {
  classifyError,
  retryOperation,
  normalizePath,
  validateFilename,
  safeFileOperation
};

Performance Optimization Strategies

Understanding when to use synchronous versus asynchronous file operations directly impacts application performance and responsiveness. Synchronous operations block the entire Node.js event loop, preventing any other code from executing until the file operation completes. For short-lived scripts or initialization code that runs before serving requests, this blocking behavior is acceptable and even preferable—it simplifies code by eliminating callback handling and ensures operations complete in a predictable order. However, using synchronous file operations in HTTP request handlers, WebSocket message processors, or any callback on the event loop creates severe performance problems, as each blocked request prevents Node.js from handling other concurrent operations.

Asynchronous operations with callbacks or promises allow Node.js to continue processing other events while waiting for disk I/O to complete. Modern SSDs complete operations in microseconds to milliseconds, but traditional hard drives can take 5-15 milliseconds for seek operations. In a server handling hundreds of requests per second, blocking for even a few milliseconds per operation destroys throughput. The promises-based API with async/await provides non-blocking behavior with readable code structure, making it the default choice for any code on the event loop. The key principle: use synchronous operations only during application startup or in worker threads dedicated to blocking I/O, never in the main event loop during request handling.

Batching file operations and parallelism tuning significantly impact performance for bulk operations. Processing thousands of files serially—awaiting each operation before starting the next—underutilizes system capabilities. Most operating systems can handle dozens or hundreds of concurrent file operations efficiently, limited by disk I/O bandwidth rather than operation count. Using Promise.all() to parallelize independent operations dramatically reduces total execution time. However, unbounded parallelism creates problems: opening thousands of files simultaneously exhausts file descriptors, and overwhelming the disk with random access patterns degrades performance below serial execution. A controlled concurrency pattern using pools or semaphores provides optimal performance by keeping the disk busy without overwhelming system resources.

const fs = require('fs').promises;
const path = require('path');

/**
 * Process files with controlled concurrency
 * Prevents overwhelming system resources
 */
async function processWithConcurrency(items, processor, concurrency = 10) {
  const results = [];
  const executing = [];

  for (const item of items) {
    // Create promise for this item
    const promise = Promise.resolve().then(() => processor(item));
    results.push(promise);

    // If concurrency limit reached, wait for one to complete
    if (concurrency <= items.length) {
      const executing = promise.then(() => {
        executing.splice(executing.indexOf(executing), 1);
      });
      executing.push(executing);

      if (executing.length >= concurrency) {
        await Promise.race(executing);
      }
    }
  }

  return Promise.all(results);
}

/**
 * Efficiently copy multiple files with concurrency control
 */
async function batchCopy(sourceDestPairs, options = {}) {
  const { concurrency = 5, onProgress = null } = options;
  let completed = 0;

  const copyOne = async (pair) => {
    try {
      await fs.copyFile(pair.source, pair.dest);
      completed++;
      
      if (onProgress) {
        onProgress({ completed, total: sourceDestPairs.length, current: pair.source });
      }

      return { success: true, ...pair };
    } catch (error) {
      return { success: false, error: error.message, ...pair };
    }
  };

  const results = await processWithConcurrency(sourceDestPairs, copyOne, concurrency);
  
  const successful = results.filter(r => r.success).length;
  const failed = results.filter(r => !r.success).length;

  return { successful, failed, results };
}

/**
 * Stream-based large file copy with progress
 * More memory efficient than buffer-based copying
 */
async function copyLargeFile(sourcePath, destPath, options = {}) {
  const { onProgress = null } = options;

  return new Promise((resolve, reject) => {
    const readStream = require('fs').createReadStream(sourcePath);
    const writeStream = require('fs').createWriteStream(destPath);

    let bytesWritten = 0;
    let sourceSize = 0;

    // Get source size for progress reporting
    fs.stat(sourcePath).then(stats => {
      sourceSize = stats.size;
    });

    readStream.on('data', (chunk) => {
      bytesWritten += chunk.length;
      
      if (onProgress && sourceSize > 0) {
        const percent = (bytesWritten / sourceSize) * 100;
        onProgress({ bytesWritten, sourceSize, percent: percent.toFixed(2) });
      }
    });

    readStream.on('error', (error) => {
      writeStream.destroy();
      reject(error);
    });

    writeStream.on('error', (error) => {
      readStream.destroy();
      reject(error);
    });

    writeStream.on('finish', () => {
      resolve({ bytesWritten, sourcePath, destPath });
    });

    readStream.pipe(writeStream);
  });
}

/**
 * Benchmark sync vs async operations
 */
async function benchmarkOperations(dirPath, operationCount = 100) {
  const results = {};

  // Benchmark synchronous readdir
  const syncStart = Date.now();
  for (let i = 0; i < operationCount; i++) {
    require('fs').readdirSync(dirPath);
  }
  results.syncDuration = Date.now() - syncStart;

  // Benchmark async readdir (serial)
  const asyncSerialStart = Date.now();
  for (let i = 0; i < operationCount; i++) {
    await fs.readdir(dirPath);
  }
  results.asyncSerialDuration = Date.now() - asyncSerialStart;

  // Benchmark async readdir (parallel)
  const asyncParallelStart = Date.now();
  await Promise.all(
    Array(operationCount).fill().map(() => fs.readdir(dirPath))
  );
  results.asyncParallelDuration = Date.now() - asyncParallelStart;

  return results;
}

module.exports = {
  processWithConcurrency,
  batchCopy,
  copyLargeFile,
  benchmarkOperations
};

Atomic Operations and Data Integrity

Ensuring file operations complete successfully without leaving the file system in inconsistent states requires understanding atomicity guarantees at different levels. Individual system calls like rename(), unlink(), and mkdir() are generally atomic—they either complete fully or have no effect—but multi-step operations combining several system calls lack atomicity without additional safeguards. Building reliable file operations on top of these primitives requires implementing patterns that minimize windows of inconsistency and provide recovery mechanisms for failures.

The write-rename pattern mentioned earlier exemplifies atomic update design. By writing to a temporary file and then renaming it over the target, applications ensure that readers either see the complete old content or the complete new content, never a partially written mixture. This pattern extends to directory operations: when updating a directory structure, create the new structure in a temporary location, then atomically rename it to the target name. For example, extracting an archive safely involves extracting to target.tmp, verifying extraction completed successfully, then renaming target.tmp to target. This approach prevents partially extracted directories from being mistaken for complete installations.

Lock files provide a mechanism for coordinating file access across processes when exclusive access is required. The typical pattern creates a lock file using fs.open() with the O_CREAT | O_EXCL flags (via wx mode), which fails atomically if the file already exists. The process holds the lock while performing non-atomic operations, then deletes the lock file upon completion. This pattern isn't foolproof—process crashes can leave stale lock files requiring manual cleanup—but it prevents common race conditions. More sophisticated locking uses flock() system calls via native modules or writes process IDs to lock files enabling stale lock detection, but these approaches trade simplicity for robustness.

const fs = require('fs').promises;
const path = require('path');

/**
 * Atomic file update with rollback capability
 */
async function atomicUpdate(filePath, updateFn, options = {}) {
  const { 
    createBackup = true,
    fsync = false 
  } = options;

  const dir = path.dirname(filePath);
  const filename = path.basename(filePath);
  const backupPath = path.join(dir, `${filename}.backup-${Date.now()}`);
  const tempPath = path.join(dir, `.${filename}.${process.pid}.tmp`);

  let originalContent = null;
  let backupCreated = false;

  try {
    // Read original content if file exists
    try {
      originalContent = await fs.readFile(filePath, 'utf-8');
      
      // Create backup if requested
      if (createBackup && originalContent !== null) {
        await fs.copyFile(filePath, backupPath);
        backupCreated = true;
      }
    } catch (error) {
      if (error.code !== 'ENOENT') {
        throw error;
      }
      // File doesn't exist, proceed with creation
    }

    // Apply update function to get new content
    const newContent = await updateFn(originalContent);

    // Write to temporary file
    await fs.writeFile(tempPath, newContent, 'utf-8');

    // Fsync if requested
    if (fsync) {
      const handle = await fs.open(tempPath, 'r+');
      try {
        await handle.sync();
      } finally {
        await handle.close();
      }
    }

    // Atomic rename
    await fs.rename(tempPath, filePath);

    // Cleanup backup on success
    if (backupCreated && !options.keepBackup) {
      await fs.unlink(backupPath).catch(() => {});
    }

    return { 
      success: true, 
      backupPath: backupCreated ? backupPath : null 
    };

  } catch (error) {
    // Cleanup temp file
    try {
      await fs.unlink(tempPath);
    } catch {
      // Ignore cleanup errors
    }

    // Restore from backup if available
    if (backupCreated && options.autoRestore) {
      try {
        await fs.copyFile(backupPath, filePath);
      } catch (restoreError) {
        throw new Error(
          `Update failed and restore failed: ${error.message}, ${restoreError.message}`
        );
      }
    }

    throw error;
  }
}

/**
 * Lock file implementation for exclusive access
 */
class FileLock {
  constructor(lockPath, options = {}) {
    this.lockPath = lockPath;
    this.timeout = options.timeout || 30000;
    this.pollInterval = options.pollInterval || 100;
    this.fileHandle = null;
  }

  /**
   * Acquire lock, waiting up to timeout
   */
  async acquire() {
    const startTime = Date.now();

    while (Date.now() - startTime < this.timeout) {
      try {
        // Try to create lock file exclusively
        this.fileHandle = await fs.open(this.lockPath, 'wx');
        
        // Write process ID for debugging
        await fs.writeFile(this.lockPath, JSON.stringify({
          pid: process.pid,
          acquired: new Date().toISOString(),
          hostname: require('os').hostname()
        }));

        return true;

      } catch (error) {
        if (error.code === 'EEXIST') {
          // Lock exists, wait and retry
          await new Promise(resolve => setTimeout(resolve, this.pollInterval));
          continue;
        }
        throw error;
      }
    }

    throw new Error(`Failed to acquire lock after ${this.timeout}ms: ${this.lockPath}`);
  }

  /**
   * Release lock
   */
  async release() {
    if (this.fileHandle) {
      await this.fileHandle.close();
      this.fileHandle = null;
    }

    try {
      await fs.unlink(this.lockPath);
    } catch (error) {
      if (error.code !== 'ENOENT') {
        console.warn(`Failed to remove lock file: ${error.message}`);
      }
    }
  }

  /**
   * Execute function with lock held
   */
  async withLock(fn) {
    await this.acquire();
    try {
      return await fn();
    } finally {
      await this.release();
    }
  }

  /**
   * Check if lock is stale (held by dead process)
   */
  async isStale() {
    try {
      const content = await fs.readFile(this.lockPath, 'utf-8');
      const lockInfo = JSON.parse(content);
      
      // Check if process is still running (Unix only)
      if (process.platform !== 'win32') {
        try {
          process.kill(lockInfo.pid, 0); // Signal 0 just checks if process exists
          return false; // Process exists, lock is not stale
        } catch {
          return true; // Process doesn't exist, lock is stale
        }
      }

      return false; // Can't determine on Windows, assume not stale
    } catch {
      return false; // Can't read lock file, assume not stale
    }
  }
}

module.exports = { atomicUpdate, FileLock };

Production Patterns and Real-World Implementations

Building production file system utilities requires combining the fundamental operations into higher-level abstractions that handle complexity while providing clean interfaces. A file synchronization utility, for instance, must compare source and destination trees, identify differences, and apply only necessary changes—a task involving listing, comparing metadata, copying new or modified files, and optionally deleting removed files. The implementation must handle large file counts efficiently, provide progress reporting for long-running operations, and recover gracefully from transient errors without restarting the entire synchronization.

Log rotation represents another common production pattern where file system operations enable system reliability. Applications typically write logs to a single file, but unbounded log growth eventually exhausts disk space. Log rotation periodically renames the current log file (e.g., app.log becomes app.log.1), potentially compressing old logs, and creates a fresh app.log for new entries. Implementing rotation safely requires handling the race between the rotation process and the application writing logs—often solved through file reopening triggered by signals, atomic renames, or copy-truncate strategies. Understanding when file descriptors remain valid after renames (on Unix) versus when they become invalid (some operations on Windows) affects rotation strategy design.

Temporary file management, while seemingly simple, involves subtle considerations around cleanup, naming, and security. Temporary files should use unpredictable names to prevent security vulnerabilities where attackers predict temp file paths and create malicious files before the legitimate process. The pattern combines a base name, process ID, timestamp, and random component to ensure uniqueness. Cleanup must handle both normal exits and crashes—using process.on('exit') for normal cleanup and potentially a periodic background task to remove orphaned temp files older than a threshold. Some applications use fs.mkdtemp() to create temporary directories with guaranteed unique names, isolating each operation's temporary files.

const fs = require('fs').promises;
const path = require('path');
const { createHash } = require('crypto');

/**
 * Synchronize directory contents from source to destination
 */
async function syncDirectory(sourceDir, destDir, options = {}) {
  const {
    deleteOrphaned = false,
    dryRun = false,
    compareContent = false,
    onProgress = null
  } = options;

  const operations = {
    copied: [],
    updated: [],
    deleted: [],
    skipped: []
  };

  // Ensure destination directory exists
  if (!dryRun) {
    await fs.mkdir(destDir, { recursive: true });
  }

  // Read source directory
  const sourceEntries = await fs.readdir(sourceDir, { withFileTypes: true });

  for (const entry of sourceEntries) {
    const sourcePath = path.join(sourceDir, entry.name);
    const destPath = path.join(destDir, entry.name);

    try {
      if (entry.isDirectory()) {
        // Recursively sync subdirectories
        const subResults = await syncDirectory(sourcePath, destPath, options);
        operations.copied.push(...subResults.copied);
        operations.updated.push(...subResults.updated);
        operations.deleted.push(...subResults.deleted);
        operations.skipped.push(...subResults.skipped);

      } else if (entry.isFile()) {
        // Check if destination exists
        let destExists = false;
        let needsUpdate = false;

        try {
          const destStats = await fs.stat(destPath);
          destExists = true;

          // Compare modification times
          const sourceStats = await fs.stat(sourcePath);
          
          if (compareContent) {
            // Compare file hashes (slower but accurate)
            needsUpdate = await filesAreDifferent(sourcePath, destPath);
          } else {
            // Compare by size and mtime (faster but may miss changes)
            needsUpdate = sourceStats.size !== destStats.size || 
                         sourceStats.mtimeMs > destStats.mtimeMs;
          }

        } catch (error) {
          if (error.code !== 'ENOENT') {
            throw error;
          }
          // Destination doesn't exist, needs copy
        }

        if (!destExists) {
          if (!dryRun) {
            await fs.copyFile(sourcePath, destPath);
          }
          operations.copied.push(destPath);
        } else if (needsUpdate) {
          if (!dryRun) {
            await fs.copyFile(sourcePath, destPath);
          }
          operations.updated.push(destPath);
        } else {
          operations.skipped.push(destPath);
        }
      }

      if (onProgress) {
        onProgress(operations);
      }

    } catch (error) {
      console.error(`Error syncing ${sourcePath}: ${error.message}`);
    }
  }

  // Handle orphaned files in destination
  if (deleteOrphaned) {
    try {
      const destEntries = await fs.readdir(destDir, { withFileTypes: true });
      const sourceNames = new Set(sourceEntries.map(e => e.name));

      for (const entry of destEntries) {
        if (!sourceNames.has(entry.name)) {
          const orphanPath = path.join(destDir, entry.name);
          
          if (!dryRun) {
            if (entry.isDirectory()) {
              await fs.rm(orphanPath, { recursive: true, force: true });
            } else {
              await fs.unlink(orphanPath);
            }
          }
          operations.deleted.push(orphanPath);
        }
      }
    } catch (error) {
      console.error(`Error checking for orphaned files: ${error.message}`);
    }
  }

  return operations;
}

/**
 * Compare files by content hash
 */
async function filesAreDifferent(path1, path2) {
  const [hash1, hash2] = await Promise.all([
    computeFileHash(path1),
    computeFileHash(path2)
  ]);

  return hash1 !== hash2;
}

/**
 * Compute SHA-256 hash of file
 */
async function computeFileHash(filePath) {
  return new Promise((resolve, reject) => {
    const hash = createHash('sha256');
    const stream = require('fs').createReadStream(filePath);

    stream.on('data', (chunk) => hash.update(chunk));
    stream.on('end', () => resolve(hash.digest('hex')));
    stream.on('error', reject);
  });
}

/**
 * Rotate log files with age-based retention
 */
async function rotateLogs(logPath, options = {}) {
  const {
    maxFiles = 5,
    compress = false
  } = options;

  try {
    // Check if current log exists
    await fs.access(logPath);
  } catch {
    // Log file doesn't exist, nothing to rotate
    return { success: true, rotated: false };
  }

  // Rotate existing numbered logs
  for (let i = maxFiles - 1; i >= 1; i--) {
    const oldPath = `${logPath}.${i}`;
    const newPath = `${logPath}.${i + 1}`;

    try {
      await fs.rename(oldPath, newPath);
    } catch (error) {
      if (error.code !== 'ENOENT') {
        throw error;
      }
      // Old log doesn't exist, skip
    }
  }

  // Rename current log to .1
  await fs.rename(logPath, `${logPath}.1`);

  // Create new empty log file
  await fs.writeFile(logPath, '', 'utf-8');

  // Compress old logs if requested
  if (compress) {
    const zlib = require('zlib').promises;
    const readStream = require('fs').createReadStream(`${logPath}.1`);
    const writeStream = require('fs').createWriteStream(`${logPath}.1.gz`);
    
    await new Promise((resolve, reject) => {
      readStream
        .pipe(require('zlib').createGzip())
        .pipe(writeStream)
        .on('finish', resolve)
        .on('error', reject);
    });

    await fs.unlink(`${logPath}.1`);
  }

  // Delete oldest log if exceeds maxFiles
  const oldestPath = `${logPath}.${maxFiles + 1}`;
  try {
    await fs.unlink(oldestPath);
  } catch {
    // File might not exist
  }

  return { success: true, rotated: true };
}

/**
 * Managed temporary file with automatic cleanup
 */
class TempFile {
  constructor(options = {}) {
    this.dir = options.dir || require('os').tmpdir();
    this.prefix = options.prefix || 'tmp-';
    this.path = null;
    this.cleanupRegistered = false;
  }

  /**
   * Create temporary file
   */
  async create() {
    // Generate unique filename
    const randomBytes = require('crypto').randomBytes(8).toString('hex');
    const filename = `${this.prefix}${process.pid}-${Date.now()}-${randomBytes}`;
    this.path = path.join(this.dir, filename);

    // Create empty file
    await fs.writeFile(this.path, '', 'utf-8');

    // Register cleanup handler
    if (!this.cleanupRegistered) {
      this.registerCleanup();
      this.cleanupRegistered = true;
    }

    return this.path;
  }

  /**
   * Write content to temp file
   */
  async write(content) {
    if (!this.path) {
      await this.create();
    }
    await fs.writeFile(this.path, content);
  }

  /**
   * Read content from temp file
   */
  async read() {
    if (!this.path) {
      throw new Error('Temp file not created');
    }
    return fs.readFile(this.path, 'utf-8');
  }

  /**
   * Delete temp file
   */
  async cleanup() {
    if (this.path) {
      try {
        await fs.unlink(this.path);
      } catch (error) {
        if (error.code !== 'ENOENT') {
          console.warn(`Failed to cleanup temp file ${this.path}: ${error.message}`);
        }
      }
      this.path = null;
    }
  }

  /**
   * Register cleanup on process exit
   */
  registerCleanup() {
    const cleanup = () => {
      if (this.path) {
        try {
          require('fs').unlinkSync(this.path);
        } catch {
          // Best effort cleanup
        }
      }
    };

    process.on('exit', cleanup);
    process.on('SIGINT', () => {
      cleanup();
      process.exit(130);
    });
    process.on('SIGTERM', () => {
      cleanup();
      process.exit(143);
    });
  }
}

module.exports = { atomicUpdate, rotateLogs, TempFile };

Monitoring and Watching File System Changes

While not strictly a manipulation operation, watching file system changes enables reactive patterns where applications respond automatically to file modifications, additions, or deletions. Node.js provides fs.watch() and fs.watchFile() for monitoring file system changes, each with different characteristics and trade-offs. The fs.watch() function uses efficient operating system primitives (inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows) that notify the application when changes occur, avoiding expensive polling. However, these OS-specific implementations create cross-platform inconsistencies—some platforms report the filename that changed, others only report that something in a directory changed.

The fs.watchFile() function provides a more portable but less efficient alternative using stat polling. It periodically calls fs.stat() on the target file and compares timestamps to detect changes. This approach works consistently across platforms and can monitor files on network file systems where native watching might not work, but it requires continuous polling that increases system load and introduces latency between change and detection. For watching a handful of files where consistency matters more than efficiency, fs.watchFile() provides reliable cross-platform behavior. For watching directories containing hundreds or thousands of files, the efficiency of fs.watch() becomes necessary despite its quirks.

Building reliable file watchers requires handling numerous edge cases: rapid successive changes (debouncing), editor save strategies that delete-then-create rather than modify in place, symbolic link changes, and watcher failures that stop reporting events silently. Production file watching typically wraps the low-level APIs with logic that normalizes events across platforms, implements debouncing to batch rapid changes, and includes health checks that verify the watcher still functions. Many applications ultimately use libraries like chokidar for production file watching because implementing robust, cross-platform watching is surprisingly complex, but understanding the underlying APIs clarifies what these libraries provide and when simpler built-in approaches suffice.

const fs = require('fs');
const path = require('path');

/**
 * Simple file watcher with debouncing
 */
class FileWatcher {
  constructor(targetPath, callback, options = {}) {
    this.targetPath = targetPath;
    this.callback = callback;
    this.debounceMs = options.debounceMs || 100;
    this.persistent = options.persistent !== false;
    this.watcher = null;
    this.debounceTimer = null;
    this.lastEvent = null;
  }

  /**
   * Start watching
   */
  start() {
    this.watcher = fs.watch(this.targetPath, { 
      persistent: this.persistent,
      recursive: false 
    }, (eventType, filename) => {
      this.handleChange(eventType, filename);
    });

    this.watcher.on('error', (error) => {
      console.error(`Watcher error for ${this.targetPath}:`, error.message);
      this.callback({ type: 'error', error, path: this.targetPath });
    });

    return this;
  }

  /**
   * Handle change with debouncing
   */
  handleChange(eventType, filename) {
    const event = {
      type: eventType,
      filename,
      path: this.targetPath,
      timestamp: Date.now()
    };

    // Clear existing debounce timer
    if (this.debounceTimer) {
      clearTimeout(this.debounceTimer);
    }

    // Set new debounce timer
    this.debounceTimer = setTimeout(() => {
      this.callback(event);
      this.lastEvent = event;
      this.debounceTimer = null;
    }, this.debounceMs);
  }

  /**
   * Stop watching
   */
  stop() {
    if (this.watcher) {
      this.watcher.close();
      this.watcher = null;
    }
    if (this.debounceTimer) {
      clearTimeout(this.debounceTimer);
      this.debounceTimer = null;
    }
  }
}

/**
 * Watch directory and report specific change types
 */
async function watchDirectoryChanges(dirPath, callback, options = {}) {
  // Take snapshot of current state
  let currentState = await captureDirectoryState(dirPath);

  const watcher = new FileWatcher(dirPath, async (event) => {
    try {
      // Capture new state
      const newState = await captureDirectoryState(dirPath);

      // Detect specific changes
      const changes = detectChanges(currentState, newState);

      // Report changes
      if (changes.added.length > 0 || changes.modified.length > 0 || changes.deleted.length > 0) {
        callback(changes);
      }

      // Update current state
      currentState = newState;

    } catch (error) {
      console.error('Error detecting changes:', error.message);
    }
  }, options);

  return watcher.start();
}

/**
 * Capture directory state for comparison
 */
async function captureDirectoryState(dirPath) {
  const state = new Map();

  try {
    const entries = await fs.promises.readdir(dirPath, { withFileTypes: true });

    for (const entry of entries) {
      if (entry.isFile()) {
        const filePath = path.join(dirPath, entry.name);
        const stats = await fs.promises.stat(filePath);
        
        state.set(entry.name, {
          size: stats.size,
          mtime: stats.mtimeMs,
          mode: stats.mode
        });
      }
    }
  } catch (error) {
    console.error(`Error capturing state: ${error.message}`);
  }

  return state;
}

/**
 * Detect changes between two directory states
 */
function detectChanges(oldState, newState) {
  const changes = {
    added: [],
    modified: [],
    deleted: []
  };

  // Detect added and modified files
  for (const [filename, newMeta] of newState.entries()) {
    if (!oldState.has(filename)) {
      changes.added.push(filename);
    } else {
      const oldMeta = oldState.get(filename);
      if (oldMeta.size !== newMeta.size || oldMeta.mtime !== newMeta.mtime) {
        changes.modified.push(filename);
      }
    }
  }

  // Detect deleted files
  for (const filename of oldState.keys()) {
    if (!newState.has(filename)) {
      changes.deleted.push(filename);
    }
  }

  return changes;
}

/**
 * Hot reload pattern: watch file and reload on change
 */
function watchAndReload(configPath, loadFn, options = {}) {
  let currentConfig = null;
  let watcher = null;

  const reload = async () => {
    try {
      const newConfig = await loadFn(configPath);
      currentConfig = newConfig;
      
      if (options.onReload) {
        options.onReload(newConfig);
      }

      console.log(`Configuration reloaded: ${configPath}`);
    } catch (error) {
      console.error(`Failed to reload configuration: ${error.message}`);
      if (options.onError) {
        options.onError(error);
      }
    }
  };

  // Initial load
  reload();

  // Watch for changes
  watcher = new FileWatcher(configPath, () => {
    reload();
  }, { debounceMs: options.debounceMs || 500 });

  watcher.start();

  return {
    getConfig: () => currentConfig,
    reload,
    stop: () => watcher.stop()
  };
}

module.exports = { FileWatcher, watchDirectoryChanges, watchAndReload };

Common Pitfalls and How to Avoid Them

One of the most insidious pitfalls in file system programming involves race conditions between checking and acting on file state. The pattern of checking whether a file exists using fs.access() or fs.stat(), then performing an operation based on that check, creates a window where another process might modify the file system state between check and action. Instead of check-then-act patterns, use atomic operations that either succeed or fail based on current state. For example, rather than checking if a file exists before creating it, use fs.open() with the wx flag (write, fail if exists) or fs.copyFile() with COPYFILE_EXCL. These operations combine the check and action atomically at the kernel level.

Path traversal vulnerabilities emerge when applications construct file paths from user input without proper validation. An attacker providing paths like ../../../etc/passwd can access files outside intended directories if the application naively concatenates paths. Always use path.resolve() to convert relative paths to absolute ones, then verify the resolved path starts with the intended base directory using path.relative() and checking the result doesn't start with ... Never trust user-provided paths without validation, even in internal tools—security vulnerabilities often emerge from unexpected input sources or misuse of internal APIs. The validation pattern from the CLI article applies equally to file system operations.

Error handling strategy significantly impacts reliability. Treating all errors identically—either always failing fast or always continuing—rarely matches actual requirements. Distinguish between transient errors that might succeed on retry (like EAGAIN or EBUSY), permanent errors where retry won't help (like ENOENT or EACCES), and errors indicating programming bugs (like EINVAL). Implement retry logic with exponential backoff for transient errors, report permission errors clearly to users, and log unexpected errors with full context for debugging. The error classification approach shown earlier enables this nuanced handling without sprawling if-else chains.

const fs = require('fs').promises;
const path = require('path');

/**
 * Safe path resolution with directory restriction
 */
function safeResolvePath(userPath, baseDir) {
  // Resolve both to absolute paths
  const resolved = path.resolve(baseDir, userPath);
  const normalizedBase = path.resolve(baseDir);

  // Check if resolved path is within base directory
  const relative = path.relative(normalizedBase, resolved);
  
  // Path is safe if relative doesn't start with .. and isn't absolute
  const isSafe = relative && 
                 !relative.startsWith('..') && 
                 !path.isAbsolute(relative);

  if (!isSafe) {
    throw new Error(
      `Path traversal attempt detected: '${userPath}' resolves outside base directory`
    );
  }

  return resolved;
}

/**
 * Atomic check-and-create operation
 * Avoids TOCTOU race condition
 */
async function createIfNotExists(filePath, content, options = {}) {
  try {
    // Open with exclusive creation flag
    // This is atomic - either creates the file or fails if it exists
    const handle = await fs.open(filePath, 'wx');
    
    try {
      await handle.writeFile(content, options.encoding || 'utf-8');
      return { created: true, existed: false };
    } finally {
      await handle.close();
    }

  } catch (error) {
    if (error.code === 'EEXIST') {
      return { created: false, existed: true };
    }
    throw error;
  }
}

/**
 * Defensive directory creation with existence handling
 */
async function ensureDirectory(dirPath, options = {}) {
  try {
    await fs.mkdir(dirPath, { 
      recursive: true,
      mode: options.mode || 0o755
    });
    return { created: true };

  } catch (error) {
    // mkdir with recursive:true should not fail if directory exists
    // but handle it anyway for older Node versions
    if (error.code === 'EEXIST') {
      // Verify it's actually a directory
      const stats = await fs.stat(dirPath);
      if (!stats.isDirectory()) {
        throw new Error(`Path exists but is not a directory: ${dirPath}`);
      }
      return { created: false, existed: true };
    }
    throw error;
  }
}

/**
 * Validate file operation won't exceed system limits
 */
async function validateSystemLimits(operation, options = {}) {
  const limits = {
    maxFileSize: options.maxFileSize || 10 * 1024 * 1024 * 1024, // 10GB default
    maxPathLength: process.platform === 'win32' ? 260 : 4096,
    maxFilenameLength: 255
  };

  if (operation.type === 'copy' || operation.type === 'write') {
    // Check source file size
    if (operation.sourcePath) {
      const stats = await fs.stat(operation.sourcePath);
      if (stats.size > limits.maxFileSize) {
        throw new Error(
          `File size ${stats.size} exceeds maximum ${limits.maxFileSize}`
        );
      }
    }

    // Check destination path length
    if (operation.destPath) {
      if (operation.destPath.length > limits.maxPathLength) {
        throw new Error(
          `Path length ${operation.destPath.length} exceeds maximum ${limits.maxPathLength}`
        );
      }

      const filename = path.basename(operation.destPath);
      if (filename.length > limits.maxFilenameLength) {
        throw new Error(
          `Filename length ${filename.length} exceeds maximum ${limits.maxFilenameLength}`
        );
      }
    }
  }

  return true;
}

/**
 * Prevent concurrent modifications using operation queue
 */
class FileOperationQueue {
  constructor() {
    this.locks = new Map();
  }

  /**
   * Execute operation exclusively for a given path
   */
  async execute(filePath, operation) {
    const normalized = path.resolve(filePath);

    // Wait for any existing operation on this path
    while (this.locks.has(normalized)) {
      await this.locks.get(normalized);
    }

    // Create lock for this operation
    const lockPromise = operation().finally(() => {
      this.locks.delete(normalized);
    });

    this.locks.set(normalized, lockPromise);

    return lockPromise;
  }

  /**
   * Get number of pending operations
   */
  pending() {
    return this.locks.size;
  }
}

module.exports = {
  safeResolvePath,
  createIfNotExists,
  ensureDirectory,
  validateSystemLimits,
  FileOperationQueue
};

Advanced Patterns: Streaming and Transformation Pipelines

For operations on large files where loading entire contents into memory is impractical, Node.js streams provide memory-efficient processing through incremental chunk handling. A streaming copy operation reads source data in chunks (typically 64KB), writes each chunk to the destination, and continues without accumulating data in memory. This approach enables copying files larger than available RAM and provides natural progress reporting opportunities—each chunk represents quantifiable progress toward completion. The pipeline() function from the stream module handles backpressure automatically, preventing fast readers from overwhelming slow writers.

Transform streams enable on-the-fly data modification during copy operations—compression, encryption, line filtering, or format conversion—without requiring temporary intermediate files. By inserting transform streams between read and write streams, applications create efficient processing pipelines that compose operations naturally. For example, a backup utility might pipe a file read stream through a compression transform stream, then through an encryption transform stream, finally writing to a destination file. Each transformation occurs incrementally on chunks, maintaining constant memory usage regardless of file size. This pipeline approach mirrors Unix shell pipe composition but within Node.js processes.

Handling large directory operations efficiently requires balancing memory usage, operation count, and progress reporting. When processing tens of thousands of files, accumulating all paths into arrays before processing consumes significant memory and delays operation start. Instead, process files as they're discovered using async iterators or stream-like patterns where directory traversal yields paths incrementally and processing begins immediately. This approach provides better progress feedback—operations start processing files within milliseconds rather than waiting for complete discovery—and uses bounded memory regardless of total file count.

const fs = require('fs');
const { pipeline } = require('stream').promises;
const { Transform } = require('stream');

/**
 * Copy file using streams with progress reporting
 */
async function streamCopy(sourcePath, destPath, options = {}) {
  const { onProgress = null } = options;

  // Get source size for progress calculation
  const stats = await fs.promises.stat(sourcePath);
  const totalSize = stats.size;
  let transferred = 0;

  // Create progress reporting transform stream
  const progressStream = new Transform({
    transform(chunk, encoding, callback) {
      transferred += chunk.length;
      
      if (onProgress) {
        onProgress({
          transferred,
          totalSize,
          percent: ((transferred / totalSize) * 100).toFixed(2)
        });
      }

      callback(null, chunk);
    }
  });

  // Create pipeline
  const source = fs.createReadStream(sourcePath);
  const destination = fs.createWriteStream(destPath);

  await pipeline(source, progressStream, destination);

  return { success: true, bytesTransferred: transferred };
}

/**
 * Copy with transformation (e.g., compression)
 */
async function copyWithCompression(sourcePath, destPath, options = {}) {
  const zlib = require('zlib');
  const source = fs.createReadStream(sourcePath);
  const compress = zlib.createGzip(options.compressionOptions);
  const destination = fs.createWriteStream(destPath);

  await pipeline(source, compress, destination);

  return { success: true, sourcePath, destPath };
}

/**
 * Process large directory with streaming approach
 * Memory-efficient for directories with thousands of files
 */
async function* streamDirectoryEntries(dirPath, options = {}) {
  const { recursive = false, filter = null } = options;

  const entries = await fs.promises.readdir(dirPath, { withFileTypes: true });

  for (const entry of entries) {
    const entryPath = path.join(dirPath, entry.name);

    // Apply filter
    if (filter && !filter(entry, entryPath)) {
      continue;
    }

    yield { entry, path: entryPath };

    // Recursively process subdirectories
    if (recursive && entry.isDirectory()) {
      yield* streamDirectoryEntries(entryPath, options);
    }
  }
}

/**
 * Example: Incremental backup with streaming
 */
async function incrementalBackup(sourceDir, backupDir, options = {}) {
  const { 
    compressionEnabled = true,
    concurrency = 5 
  } = options;

  // Ensure backup directory exists
  await fs.promises.mkdir(backupDir, { recursive: true });

  let backed = 0;
  let skipped = 0;
  const operations = [];

  // Stream through source directory
  for await (const { entry, path: sourcePath } of streamDirectoryEntries(sourceDir, { recursive: true })) {
    if (!entry.isFile()) continue;

    const relativePath = path.relative(sourceDir, sourcePath);
    const destPath = path.join(backupDir, relativePath);
    const destDir = path.dirname(destPath);

    // Ensure destination directory exists
    await fs.promises.mkdir(destDir, { recursive: true });

    // Determine if file needs backup
    let needsBackup = false;
    try {
      const [sourceStats, destStats] = await Promise.all([
        fs.promises.stat(sourcePath),
        fs.promises.stat(destPath)
      ]);

      needsBackup = sourceStats.mtimeMs > destStats.mtimeMs;
    } catch {
      // Destination doesn't exist
      needsBackup = true;
    }

    if (needsBackup) {
      operations.push(async () => {
        if (compressionEnabled) {
          await copyWithCompression(sourcePath, `${destPath}.gz`);
        } else {
          await streamCopy(sourcePath, destPath);
        }
        backed++;
      });
    } else {
      skipped++;
    }

    // Process in batches to control concurrency
    if (operations.length >= concurrency) {
      await Promise.all(operations.splice(0, concurrency).map(op => op()));
    }
  }

  // Process remaining operations
  if (operations.length > 0) {
    await Promise.all(operations.map(op => op()));
  }

  return { backed, skipped };
}

module.exports = {
  streamCopy,
  copyWithCompression,
  streamDirectoryEntries,
  incrementalBackup
};

Conclusion

Mastering Node.js file system operations requires understanding not just the API functions but the underlying system behaviors, edge cases, and cross-platform differences that affect reliability. The five core operations—listing, copying, moving, renaming, and deleting—combine with error handling, atomicity patterns, and performance optimization to enable building robust file manipulation tools. Modern Node.js has significantly improved its file system APIs, adding promise-based interfaces, recursive directory copying, and enhanced directory deletion, but the foundational concepts remain constant: understand synchronous versus asynchronous trade-offs, handle errors appropriately based on their type, use atomic operations where possible, and design for the inevitable failures that come with distributed timing and concurrent access.

The patterns explored in this article—from controlled concurrency and streaming operations to atomic updates and safe deletion—represent distilled wisdom from decades of file system programming across Unix, Linux, Windows, and other operating systems. These patterns transcend specific Node.js APIs; they reflect fundamental properties of how file systems work, the guarantees operating systems provide, and the failure modes that emerge from concurrent access. Whether building deployment scripts, backup utilities, build tools, or content management systems, these patterns provide a foundation for reliable file operations. The code samples demonstrate realistic implementations suitable for production use, not just toy examples, because professional software engineering demands considering the edge cases, error conditions, and performance characteristics that distinguish robust systems from fragile ones.

Key Takeaways

1. Choose the right API style for your context: Use the promises-based fs.promises API with async/await for event loop code, synchronous APIs only for startup or dedicated worker threads, and understand that mixing them inappropriately destroys application performance.
2. Use withFileTypes for efficient directory reading: When listing directories and checking entry types, the withFileTypes option returns Dirent objects that include type information without requiring separate stat() calls, dramatically improving performance for large directories.
3. Implement atomic operations using write-rename: For critical file updates, write new content to a temporary file, call fsync() to ensure disk persistence, then atomically rename to the target path, ensuring readers never observe partial writes.
4. Handle cross-device moves explicitly: When fs.rename() fails with EXDEV, implement the move as copy-verify-delete with proper error handling and temp file cleanup to ensure reliable cross-file system moves.
5. Control concurrency for batch operations: Process multiple files with bounded parallelism using Promise.all() on chunks rather than serial execution or unbounded parallelism, keeping disk busy without exhausting file descriptors or overwhelming system resources.

80/20 Insight

The 20% of file operations that handle 80% of practical needs:

Most real-world file system tasks reduce to four patterns: reading directory contents with fs.readdir(), copying individual files with fs.copyFile(), deleting with proper error handling for missing files, and ensuring directories exist with fs.mkdir(dirPath, { recursive: true }). These four operations, combined with path.join() for safe path construction and try-catch blocks that distinguish ENOENT from other errors, cover the vast majority of file manipulation needs in build scripts, deployment tools, and automation utilities.

Focus on mastering these fundamental operations with proper error handling before exploring advanced patterns like atomic updates, file watching, or streaming transforms. Many developers reach for complex recursive deletion or directory synchronization utilities when their actual needs could be satisfied by listing a directory, filtering the results, and calling delete on each matching entry. The simplest working solution—even if less elegant than sophisticated abstractions—often proves more maintainable and debuggable in production. Add complexity only when requirements genuinely demand it.

Analogies & Mental Models

The Moving Company Analogy: Understanding fs.rename() versus fs.copyFile()-then-fs.unlink() resembles the difference between a moving company relocating your furniture across town versus across the country. Moving within the same city (same file system), they can simply drive your furniture from one house to another in a single trip—fast and atomic. But moving across the country (cross-file system) requires loading everything onto a truck, driving thousands of miles, unloading, then selling or disposing of the original house contents. The cross-country move has many more failure points (truck breakdown, weather delays, damaged items) and can't be completed atomically. File moves across file systems face identical complexities.

The Restaurant Order Mental Model: Think of file system operations like restaurant orders where the kitchen (disk) processes requests from multiple waiters (processes/threads). Synchronous operations are like a waiter standing at the kitchen window blocking all other waiters until their order completes—simple for the waiter but catastrophic for restaurant throughput. Asynchronous operations are like giving orders to the kitchen and continuing to serve other tables while waiting—complex coordination but essential for serving many customers. Controlled concurrency is like limiting how many waiters can be at the kitchen window simultaneously—too few underutilizes the kitchen, too many creates chaos, the right number maximizes throughput.

References

1. Node.js File System Documentation: Official documentation for the fs module covering all APIs, options, and platform-specific behaviors. https://nodejs.org/api/fs.html

2. Node.js Path Module Documentation: Official documentation for cross-platform path manipulation and resolution. https://nodejs.org/api/path.html

3. Node.js Stream Documentation: Documentation for readable, writable, and transform streams used in efficient file processing. https://nodejs.org/api/stream.html

4. POSIX.1-2017 Standard: IEEE standard defining file system interfaces, error codes, and semantics for Unix-like systems. https://pubs.opengroup.org/onlinepubs/9699919799/

5. Linux Programmer's Manual: System call documentation including open(), rename(), unlink(), and other file operations. man 2 open, man 2 rename, etc.

6. Windows File System Documentation: Microsoft documentation on Windows file system behavior, reserved names, and path length limits. https://docs.microsoft.com/en-us/windows/win32/fileio/

7. The Node.js fs-extra Library Source Code: While not used in this article's examples, studying fs-extra's implementation provides insights into production-grade file operation patterns. https://github.com/jprichardson/node-fs-extra

8. "Stream Handbook" by James Halliday (substack): Comprehensive guide to Node.js streams including transform streams and pipeline patterns. https://github.com/substack/stream-handbook

9. "Node.js Design Patterns" by Mario Casciaro and Luciano Mammino: Published by Packt, covers asynchronous patterns, streams, and file system operations in depth. Third Edition, 2020.

10. CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition: Common Weakness Enumeration entry explaining TOCTOU vulnerabilities and mitigation strategies. https://cwe.mitre.org/data/definitions/367.html

11. Node.js fs Promises API: Documentation specific to the promises-based file system API introduced in Node.js 10. https://nodejs.org/api/fs.html#promises-api

12. libuv Documentation: The cross-platform asynchronous I/O library underlying Node.js file system operations, explaining platform abstraction. https://docs.libuv.org/