How to work with large byte array?

Question

I have a large byte array and I am splitting it into multiple byte array by 1000 bytes and process them. I am using IEnumerable and I can use foreach loop but i want to know that which number of array inside IEnumerable, I am using. I can get the total count but do not know which number of byte array I am processing, so can anyone help me to figure out how to achieve this? Please let me know if you need more details.

 public static IEnumerable<byte[]> SplitSourceBytes(byte[] SrcBytes, int size) { int srcLenght = SrcBytes.Length; byte[] source = null; int i = 0; for (; srcLenght > (i + 1) * size; i++) { source = new byte[size]; Array.Copy(SrcBytes, i * size, source, 0, size); yield return source; } int sourceLeft = srcLenght - i * size; if (sourceLeft > 0) { source = new byte[sourceLeft]; Array.Copy(SrcBytes, i * size, source, 0, sourceLeft); yield return source; } }

If you need to know what index you are on, use a for loop, not a foreach. — cadrell0
– cadrell0, Commented Nov 26, 2012 at 21:46
By number of byte array do you mean the ith chunk of the original array? You could use a dictionary to identify the chunks. — MGZero
– MGZero, Commented Nov 26, 2012 at 21:47
What do you gain by splitting the large byte array in smaller chunks? The memory allocation for the large array has already been done, and you're now allocating more memory (for the small chunks). If you want to process the large byte array in small chunks, you can consider returning an enumerable of ArraySegment<byte>, for which you have an offset where it starts on the larger array (from which you can even deduce the "index") — carlosfigueira
– carlosfigueira, Commented Nov 26, 2012 at 21:55

carlosfigueira · Accepted Answer · 2012-11-26 22:35:03Z

You don't need to create new byte arrays; you can use an ArraySegment<byte> enumeration to "split" your large array without incurring the penalty of new byte array allocations.

public static IEnumerable<ArraySegment<byte>> SplitSourceBytes(byte[] SrcBytes, int size) { int srcLength = SrcBytes.Length; int alreadyReturned = 0; while (alreadyReturned < srcLength) { int count = Math.Min(srcLength - alreadyReturned, size); yield return new ArraySegment<byte>(SrcBytes, alreadyReturned, count); alreadyReturned += count; } }

And to use it, the code would look something like the one below:

foreach (var segment in SplitSourceBytes(bytes, 1000)) { int maxAttempts = 3; int attempts = 0; bool dataSent = false; while (!dataSent && attempts < maxAttempts) { try { serverStream.Write(segment.Array, segment.Offset, segment.Count); dataSent = true; } catch (Exception ex) { // Error, try to retransmit attempts++; } } }

sa_ddam213 · Accepted Answer · 2012-11-26 22:03:19Z

This may work for what you need

 private Dictionary<int,byte[]> SplitByteArray(byte[] SrcBytes, int size) { var result = new Dictionary<int, byte[]>(); if (SrcBytes.Length > size) { int position = SrcBytes.Length; int index = 0; for (int i = 0; i < SrcBytes.Length; i += size) { int length = Math.Min(size, position); byte[] buffer = new byte[length]; Buffer.BlockCopy(SrcBytes, i, buffer, 0, length); result.Add(index, buffer); position -= size; index++; } } else { result.Add(0, SrcBytes); } return result; }

Loscas · Accepted Answer · 2012-11-26 22:49:14Z

You could place the arrays in a List that provides a count function.

public static IEnumerable<byte[]> SplitSourceBytes(byte[] SrcBytes, int size) { List<byte[]> Resultset = new List<byte[]>(); int srcLenght = SrcBytes.Length; byte[] source = null; int i = 0; for (; srcLenght > (i + 1) * size; i++) { Debug.WriteLine(string.Format("Working on byte array {0}.", Resultset.Count + 1); source = new byte[size]; Array.Copy(SrcBytes, i * size, source, 0, size); Resultset.Add(source); } int sourceLeft = srcLenght - i * size; if (sourceLeft > 0) { source = new byte[sourceLeft]; Array.Copy(SrcBytes, i * size, source, 0, sourceLeft); Resultset.Add(source); } return Resultset; }

Collectives™ on Stack Overflow

How to work with large byte array?

3 Answers 3

Comments

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Related