Large data downloads from a Web app to Android client getting Out of memory error

  • Replies:13
  • Answered
3
Robert Uomini
  • Forum posts: 10

Apr 4, 2012, 6:07:07 AM via Website

I'm sending the following request to a Web app running on Tomcat:
1connection = (HttpsURLConnection) url.openConnection();
2
3writer = new BufferedWriter(new OutputStreamWriter(connection.getOutputStream()));
4writer.write("email_addr=" + URLEncoder.encode(email_addr, "UTF-8") + "&" + "passwd=" +
5 URLEncoder.encode(password, "UTF-8") + "&" +
6 "cmd=" + tmp + "&" + "parms=" + URLEncoder.encode(parms, "UTF-8"));
7writer.close();

and the read the response as follows:

1rdr = new BufferedReader(new InputStreamReader(connection.getInputStream()));
2tmp = rdr.readLine();

Everything works fine as long as the data coming back is less than around 5 MB. Then I get an error on readLine():

04-02 16:51:34.232: E/dalvikvm-heap(3907): Out of memory on a 10309582-byte allocation

Anyone have any ideas on why this is happening and how to fix it?

Reply
31
Eric McBride
  • Forum posts: 1,790

Apr 4, 2012, 6:07:40 PM via Website

Hmm...any devs around? :)

Have you also posted this on XDA Developers?

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 4, 2012, 8:17:39 PM via Website

Eric McBride
Hmm...any devs around? :)
Guess not!

Have you also posted this on XDA Developers?
No. What is XDA?

Reply
Jeremiah 20
Jeremiah
  • Forum posts: 775

Apr 5, 2012, 2:12:51 AM via Website

Your android may have a gigabyte or more of memory, but each app is limited to a 16mb heap, 24mb on high res devices. So you could actually be running out of memory.

What are you wanting to do with the information being sent to the android? You could read a few megabytes at a time and write them to a file on sd card would be one solution, depending on what you actually need to do with the information.

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 5, 2012, 3:55:29 AM via Website

Jeremiah
Your android may have a gigabyte or more of memory, but each app is limited to a 16mb heap, 24mb on high res devices. So you could actually be running out of memory.

What are you wanting to do with the information being sent to the android? You could read a few megabytes at a time and write them to a file on sd card would be one solution, depending on what you actually need to do with the information.

Actually, this is what I've done as a workaround, and although I can save the entire file and read the whole thing into a byte array, when I try to convert the array to a String, it fails the same way.

Reply
Jeremiah 20
Jeremiah
  • Forum posts: 775

Apr 5, 2012, 5:15:10 AM via Website

Do you need all of the data in memory at once? What are you doing with the data? Could you process it a few megabytes at a time?

Another way to do it if you absolutely need to process it all at once, would be to read the data from the buffered reader a few megabytes at a time, then store it in an SQL database on the phones SD or internal memory. Once you have read in all the data and stored it in the SQL database, you can do querys and searches on it without loading it all in the ram at once.

It would help if I knew what exactly you need to do with the data?

Reply
Jeremiah 20
Jeremiah
  • Forum posts: 775

Apr 5, 2012, 5:43:32 AM via Website

You could also try converting the byte array into a string in chunks... The problem with reading the whole amount of data into a byte array and converting it into a string all at once, is that you have 2 copies of the data in memory at once. You could use a byte array input stream to load parts of the byte array ( http://developer.android.com/reference/java/io/ByteArrayInputStream.html ) and convert each part and add it to your string. You have to be careful not to create a new byte array each time, since then you would still be loading 2 copies of the data at once, instead of reusing a single small byte array.

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 5, 2012, 4:12:29 PM via Website

Jeremiah
Do you need all of the data in memory at once? What are you doing with the data? Could you process it a few megabytes at a time?

Another way to do it if you absolutely need to process it all at once, would be to read the data from the buffered reader a few megabytes at a time, then store it in an SQL database on the phones SD or internal memory. Once you have read in all the data and stored it in the SQL database, you can do querys and searches on it without loading it all in the ram at once.

It would help if I knew what exactly you need to do with the data?
I'm modifying K9 Mail to enable dynamic content (see http://www.chiaramail.com). This means that mail attachments, instead of being a part of the mail message itself, are transferred to the client from a Web server when the user wishes to view or save them. Except for this memory problem, the concept works great and implementation is almost done.

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 5, 2012, 4:34:43 PM via Website

Jeremiah
You could also try converting the byte array into a string in chunks... The problem with reading the whole amount of data into a byte array and converting it into a string all at once, is that you have 2 copies of the data in memory at once. You could use a byte array input stream to load parts of the byte array ( http://developer.android.com/reference/java/io/ByteArrayInputStream.html ) and convert each part and add it to your string. You have to be careful not to create a new byte array each time, since then you would still be loading 2 copies of the data at once, instead of reusing a single small byte array.
I tried what you suggested (I think) and ended up with the following code:

1rdr = new BufferedReader(new InputStreamReader(connection.getInputStream()));
2 tmp = "";
3
4 char[] buf = new char[MAX_SIZE];
5
6 int bytesRead = 0;
7
8 while (bytesRead >= 0) {
9 bytesRead = rdr.read(buf, 0, MAX_SIZE);
10 if (bytesRead > -1) tmp += (new String(buf)).substring(0, bytesRead);
11 }

Same problem.

Reply
31
Eric McBride
  • Forum posts: 1,790

Apr 5, 2012, 4:48:18 PM via Website

Jeremiah to the resecue!!! :)

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 5, 2012, 6:52:45 PM via Website

Not really, although I appreciate Jeremiah's help. The Out of memory error is still there.

Reply
Jeremiah 20
Jeremiah
  • Forum posts: 775

Apr 5, 2012, 6:59:34 PM via Website

Robert Uomini
Jeremiah
You could also try converting the byte array into a string in chunks... The problem with reading the whole amount of data into a byte array and converting it into a string all at once, is that you have 2 copies of the data in memory at once. You could use a byte array input stream to load parts of the byte array ( http://developer.android.com/reference/java/io/ByteArrayInputStream.html ) and convert each part and add it to your string. You have to be careful not to create a new byte array each time, since then you would still be loading 2 copies of the data at once, instead of reusing a single small byte array.
I tried what you suggested (I think) and ended up with the following code:

1rdr = new BufferedReader(new InputStreamReader(connection.getInputStream()));
2 tmp = "";
3
4 char[] buf = new char[MAX_SIZE];
5
6 int bytesRead = 0;
7
8 while (bytesRead >= 0) {
9 bytesRead = rdr.read(buf, 0, MAX_SIZE);
10 if (bytesRead > -1) tmp += (new String(buf)).substring(0, bytesRead);
11 }

Same problem.

How big is MAX_SIZE? Try setting MAX_SIZE to a lower value, it may be that as your nearing the end of available memory the "chunk" your reading is too big to fit in memory.

Though you still could run out of memory with large attachments. Correct me if I'm wrong, your trying to download the user's mail messages as an attachment and let the view their messages? Or are you trying to download attachments/files? If your running out of memory viewing messages, you could save them to an sql database and allow them to search/view a small number of messages at one time, instead of all at once.

— modified on Apr 5, 2012, 7:11:07 PM

Reply
3
Robert Uomini
  • Forum posts: 10

Apr 10, 2012, 7:18:04 AM via Website

Jeremiah
Robert Uomini
Jeremiah
You could also try converting the byte array into a string in chunks... The problem with reading the whole amount of data into a byte array and converting it into a string all at once, is that you have 2 copies of the data in memory at once. You could use a byte array input stream to load parts of the byte array ( http://developer.android.com/reference/java/io/ByteArrayInputStream.html ) and convert each part and add it to your string. You have to be careful not to create a new byte array each time, since then you would still be loading 2 copies of the data at once, instead of reusing a single small byte array.
I tried what you suggested (I think) and ended up with the following code:

1rdr = new BufferedReader(new InputStreamReader(connection.getInputStream()));
2 tmp = "";
3
4 char[] buf = new char[MAX_SIZE];
5
6 int bytesRead = 0;
7
8 while (bytesRead >= 0) {
9 bytesRead = rdr.read(buf, 0, MAX_SIZE);
10 if (bytesRead > -1) tmp += (new String(buf)).substring(0, bytesRead);
11 }

Same problem.

How big is MAX_SIZE? Try setting MAX_SIZE to a lower value, it may be that as your nearing the end of available memory the "chunk" your reading is too big to fit in memory.

Though you still could run out of memory with large attachments. Correct me if I'm wrong, your trying to download the user's mail messages as an attachment and let the view their messages? Or are you trying to download attachments/files? If your running out of memory viewing messages, you could save them to an sql database and allow them to search/view a small number of messages at one time, instead of all at once.
What I ended up doing was saving the inbound data to a file, then referencing it with the viewer via a Uri pointing to the file, which is pretty much what you suggested earlier. I had to do a bit of tweaking the existing code, since it used to expect the actual String data. Thanks for your suggestion, Jeremiah.

Reply
Jeremiah 20
Jeremiah
  • Forum posts: 775

Apr 10, 2012, 6:13:09 PM via Website

No problem Robert, glad you found the solution.

Reply