Reading and Processing Files Line by Line in C++

Processing files line by line is a common task in systems programming, data processing, and log analysis. C++ provides several approaches with different trade-offs in performance, memory usage, and code clarity.

Using std::getline with std::ifstream

The most straightforward approach uses std::getline with an input file stream:

#include <fstream>
#include <string>
#include <iostream>

int main() {
    std::ifstream file("input.txt");

    if (!file.is_open()) {
        std::cerr << "Failed to open file\n";
        return 1;
    }

    std::string line;
    while (std::getline(file, line)) {
        // Process line
        std::cout << line << "\n";
    }

    file.close();
    return 0;
}

This approach automatically handles newline characters and stops at EOF. Always check is_open() before processing to catch file access errors early.

Handling Different Line Endings

Cross-platform compatibility requires handling different line ending conventions:

#include <fstream>
#include <string>

std::string trim_line(const std::string& line) {
    size_t end = line.find_last_not_of("\r\n");
    if (end == std::string::npos) {
        return "";
    }
    return line.substr(0, end + 1);
}

int main() {
    std::ifstream file("input.txt");
    std::string line;

    while (std::getline(file, line)) {
        line = trim_line(line);
        // Process cleaned line
    }

    return 0;
}

On Windows and older Mac files, you may encounter \r\n or \r line endings. The trim_line function removes these cleanly without relying on platform-specific behavior.

Processing Large Files with Buffering

For large files, explicit buffer management improves performance:

#include <fstream>
#include <string>
#include <vector>

int main() {
    std::ifstream file("large_file.txt");
    file.rdbuf()->pubsetbuf(nullptr, 65536);  // 64KB buffer

    std::string line;
    while (std::getline(file, line)) {
        // Process line
    }

    return 0;
}

The buffer size depends on your system and file characteristics. For typical text processing, 64KB is a good starting point. Profile with perf or similar tools if throughput is critical.

Using fgets for C-style Processing

While less idiomatic in modern C++, fgets from <cstdio> can be faster for raw character processing:

#include <cstdio>
#include <cstring>

int main() {
    FILE* file = fopen("input.txt", "r");
    if (!file) {
        perror("fopen");
        return 1;
    }

    char line[1024];
    while (fgets(line, sizeof(line), file)) {
        // Remove newline if present
        size_t len = strlen(line);
        if (len > 0 && line[len - 1] == '\n') {
            line[len - 1] = '\0';
        }

        // Process line
    }

    fclose(file);
    return 0;
}

This approach works when you know your maximum line length in advance and need predictable stack allocation. Be cautious with buffer overflow—always validate line length against your buffer size.

Processing with Custom Delimiters

When lines use non-standard delimiters:

#include <fstream>
#include <string>

int main() {
    std::ifstream file("input.csv");
    std::string line;

    // Process comma-delimited file with newlines inside quoted fields
    while (std::getline(file, line, '\n')) {
        if (line.back() == '\r') {
            line.pop_back();
        }

        // Further processing for quoted fields...
    }

    return 0;
}

For CSV or other structured formats, consider using dedicated parsing libraries like mio for memory-mapped I/O or csv2 for robust CSV handling rather than reinventing delimiter parsing.

Error Handling Best Practices

Always validate file state during processing:

#include <fstream>
#include <string>
#include <iostream>

int main() {
    std::ifstream file("input.txt");

    if (!file) {
        std::cerr << "Cannot open file\n";
        return 1;
    }

    std::string line;
    int line_num = 0;

    while (std::getline(file, line)) {
        ++line_num;

        if (file.fail() && !file.eof()) {
            std::cerr << "Read error at line " << line_num << "\n";
            return 1;
        }

        // Process line
    }

    return 0;
}

Check fail() to distinguish between genuine I/O errors and normal EOF conditions. This catches disk errors or corrupted data mid-stream.

Performance Comparison

For most use cases, std::getline with default buffering is sufficient. Use C-style fgets if profiling shows I/O as your bottleneck. Memory-mapped files (mmap) are faster for random access patterns but overkill for sequential line processing.

Always profile with real data on your target hardware. I/O performance varies significantly based on disk type, file size, and system load.