c# How to run a application faster

Question

I am creating a word list of possible uppercase letters to prove how insecure 8 digit passwords are this code will write aaaaaaaa to aaaaaaab to aaaaaaac etc. until zzzzzzzz using this code:

class Program { static string path; static int file = 0; static void Main(string[] args) { new_file(); var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]"; var q = alphabet.Select(x => x.ToString()); int size = 3; int counter = 0; for (int i = 0; i < size - 1; i++) { q = q.SelectMany(x => alphabet, (x, y) => x + y); } foreach (var item in q) { if (counter >= 20000000) { new_file(); counter = 0; } if (File.Exists(path)) { using (StreamWriter sw = File.AppendText(path)) { sw.WriteLine(item); Console.WriteLine(item); /*if (!(Regex.IsMatch(item, @"(.)\1"))) { sw.WriteLine(item); counter++; } else { Console.WriteLine(item); }*/ } } else { new_file(); } } } static void new_file() { path = @"C:\" + "list" + file + ".txt"; if (!File.Exists(path)) { using (StreamWriter sw = File.CreateText(path)) { } } file++; } }

The Code is working fine but it takes Weeks to run it. Does anyone know a way to speed it up or do I have to wait? If anyone has a idea please tell me.

For speed up (i.e. propose a better algorithm) we have to know what you're trying to achieve (we hate doing reverse engeneering) — Dmitrii Bychenko
– Dmitrii Bychenko, Commented May 8, 2015 at 14:53
It is extremely easy to find problem that will take long time... Like generate all permutations of 60+ elements with length up to 10+. If you want help please make sure to cleanup your sample so only slow part is demonstrated. Use Stopwatch class to measure if you don't have access to profler. — Alexei Levenkov
– Alexei Levenkov, Commented May 8, 2015 at 14:54
Writing to the console is relatively slow, combined with the number of permutations this could take a long time.... — Polyfun
– Polyfun, Commented May 8, 2015 at 15:00
You're opening and closing the file on every iteration of the loop. Find a way to open it outside the loop and just keep writing to it until you're ready to switch files. — Craig W.
– Craig W., Commented May 8, 2015 at 15:03

Serj-Tm · Accepted Answer · 2015-05-08 16:29:58Z

Performance:

size 3: 0.02s size 4: 1.61s size 5: 144.76s

Hints:

removed LINQ for combination generation
removed Console.WriteLine for each password
removed StreamWriter
large buffer (128k) for file writing

 const string alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]"; var byteAlphabet = alphabet.Select(ch => (byte)ch).ToArray(); var alphabetLength = alphabet.Length; var newLine = new[] { (byte)'\r', (byte)'\n' }; const int size = 4; var number = new byte[size]; var password = Enumerable.Range(0, size).Select(i => byteAlphabet[0]).Concat(newLine).ToArray(); var watcher = new System.Diagnostics.Stopwatch(); watcher.Start(); var isRunning = true; for (var counter = 0; isRunning; counter++) { Console.Write("{0}: ", counter); Console.Write(password.Select(b => (char)b).ToArray()); using (var file = System.IO.File.Create(string.Format(@"list.{0:D5}.txt", counter), 2 << 16)) { for (var i = 0; i < 2000000; ++i) { file.Write(password, 0, password.Length); var j = size - 1; for (; j >= 0; j--) { if (number[j] < alphabetLength - 1) { password[j] = byteAlphabet[++number[j]]; break; } else { number[j] = 0; password[j] = byteAlphabet[0]; } } if (j < 0) { isRunning = false; break; } } } } watcher.Stop(); Console.WriteLine(watcher.Elapsed); }

@JohnOdom [Humour_mode] 3 hours then the disk space is over [/Humour_mode]
@DarkGray just one more thing i edited your code by this: if (!(Regex.IsMatch(Encoding.ASCII.GetString(password), @"(.)\1"))){ file.Write(password, 0, password.Length); } But now it is 27 times slower do you have a better method?

Craig W. · Accepted Answer · 2015-05-08 15:16:18Z

Try the following modified code. In LINQPad it runs in < 1 second. With your original code I gave up after 40 seconds. It removes the overhead of opening and closing the file for every WriteLine operation. You'll need to test and ensure it gives the same results because I'm not willing to run your original code for 24 hours to ensure the output is the same.

class Program { static string path; static int file = 0; static void Main(string[] args) { new_file(); var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]"; var q = alphabet.Select(x => x.ToString()); int size = 3; int counter = 0; for (int i = 0; i < size - 1; i++) { q = q.SelectMany(x => alphabet, (x, y) => x + y); } StreamWriter sw = File.AppendText(path); try { foreach (var item in q) { if (counter >= 20000000) { sw.Dispose(); new_file(); counter = 0; } sw.WriteLine(item); Console.WriteLine(item); } } finally { if(sw != null) { sw.Dispose(); } } } static void new_file() { path = @"C:\temp\list" + file + ".txt"; if (!File.Exists(path)) { using (StreamWriter sw = File.CreateText(path)) { } } file++; } }

counter is never getting updated, so it's all going into 1 file. That's not bad but it renders the int variable and that if statement useless.
@JohnOdom: Agreed, but I wasn't going to fix every bug in his original code (the line that increments counter was commented out in the original and I just removed that chunk from the modified version for brevity).
Okay, just wanted to let you know :P. Also is it possible to use LINQ to concatenate each item into 1 string separated with newlines so that it could all be written into 1 file in 1 go (Sounds like overkill)? Or could File.WriteAllLines be used in this case? I know I'm asking silly questions lol. But it seems like the "time consumer" is that foreach loop where it's writing each line.
This is clearly faster, but the OP's question, despite the code, was about the size=8 case. That case would still never finish (at least this single file example might crash out at an os filesystem size limit before it ate all his disk space)

tolanj · Accepted Answer · 2015-05-08 15:49:02Z

your alphabet is missing 0

With that fixed there would be 89 chars in your set. Let's call it 100 for simplicity. The set you are looking for is all the 8 character length strings drawn from that set. There are 100^8 of these, i.e. 10,000,000,000,000,000.

The disk space they will take up depends on how you encode them, lets be generous - assume you use some 8 bit char set that contains the these characters, and you don't put in carriage returns, so one byte per char, so 10,000,000,000,000,000 bytes =~ 10 peta byes?

Do you have 10 petabytes of disk? (10000 TB)?

[EDIT] In response to 'this is not an answer':

The original motivation is to create the list? The shows how large the list would be. Its hard to see what could be DONE with the list if it was actualised, i.e. it would always be quicker to reproduce it than to load it. Surely whatever point could be made by producing the list can also be made by simply knowing it's size, which the above shows how to work it out.

There are LOTS of inefficiencies in you code, but if your questions is 'how can i quickly produce this list and write it to disk' the answer is 'you literally cannot'.

[/EDIT]

This isn't an answer. It should have been posted as a comment.
Editing your post to respond to a comment isn't appropriate SO etiquette. And I'm not the OP, so it isn't my code. And finally, the question was, "How can I speed up this code" and no matter how you slice it or try to justify it, your "answer" isn't answering that question. The two downvotes you've gotten (so far) should tell you that. Have a nice day.
'Editing your post to respond to a comment isn't appropriate SO etiquette' => interesting, I was trying to respond to the comment by clarifying the answer, which is seemingly normal SO behavior. I realise I've got downvotes, I do think though this might be of interest to the OP, there are several ways of viewing this question 'Why is this code slow' OR 'I am trying to speed this up as a route to my stated goal' whence answers on the stated goal do seem appropriate. Have a nice day too.

Collectives™ on Stack Overflow

c# How to run a application faster

3 Answers 3

4 Comments

4 Comments

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

4 Comments

3 Comments

Related