Read text file with lots of line in C#

I have a text file, that could potentially have up to 1 Million lines in it, and I have a code for reading the file one line at a time, but this is taking a lot of time...lots and lots of time. Is there a method in c# to potentially optimize this process, and improve the reading. This is the code I'm using.

using(var file = new StreamReader(filePath))
{
    while((line = file.ReadLine()) != null)
     {
         //do something.
     }
}

Any suggestions on reading these lines in bulk or improving the process?

Thanks.

Thanks for all your comments. The issue had to do with the \do something where I was using the SmartXls library to write to Excel, which was causing the bottle neck. I have contacted the developers to address the issue. All the suggested solutions will work in other scenarios.

Jon Skeet
people
quotationmark

Well, this code would be simpler, if you're using .NET 4 or later you can use File.ReadLines:

foreach (var line in File.ReadLines())
{
    // Do something
}

Note that this is not the same as ReadAllLines, as ReadLines returns an IEnumerable<string> which reads lines lazily, instead of reading the whole file in one go.

The effect at execution time will be broadly the same as your original code (it won't improve performance) - this is just simpler to read.

Fundamentally, if you're reading a large file, that can take a long time - but reading just a million lines shouldn't take "lots and lots of time". My guess is that whatever you're doing with the lines takes a long time. You might want to parallelize that, potentially using a producer/consumer queue (e.g. via BlockingCollection) or TPL Dataflow, or just use Parallel LINQ, Parallel.ForEach etc.

You should use a profiler to work out where the time is being spent. If you're reading from a very slow file system, then it's possible that it really is the reading which is taking the time. We don't have enough information to guide you on that, but you should be able to narrow it down yourself.

people

See more on this question at Stackoverflow