Login or Sign Up to become a member!
LessThanDot Site Logo

LessThanDot

A Technical Community for IT Professionals

Less Than Dot is a community of passionate IT professionals and enthusiasts dedicated to sharing technical knowledge, experience, and assistance. Inside you will find reference materials, interesting technical discussions, and expert tips and commentary. Once you register for an account you will have immediate access to the forums and all past articles and commentaries.

LTD Social Sitings

Lessthandot twitter Lessthandot Linkedin Lessthandot facebook Lessthandot rss

Note: Watch for social icons on posts by your favorite authors to follow their postings on these and other social sites.

Highly Rated Users

Forum
No Posts Rated

Top 50
Given
Received

Links

Wiki
Blog

Forum Statistics

Users
Members:
1873
Members Online:
2
Guests Online:
82

Total Post History
Posts:
81445
Topics:
18714

7-Day Post History
New Posts:
0
New Topics:
0
Active Topics:
0

Our newest member
BitogClogs

Other

FAQ
All times are UTC [ DST ]

Google Ads

Out of string space error

Please wait...

Out of string space error

Postby Chopstik on Fri Oct 23, 2009 4:30 pm

Currently, we have an application that essentially creates an HTML report by returning the data from the database and then (using a StringBuilder class in VB) creates the HTML code by looping through the recordset to return each row of data. It works fine for most of what we have but we've now encountered a problem where it appears that so much data is being returned in one specific instance that it appears we're now exceeding the length of a string variable.

Because of some other constraints, I cannot simply break it up into multiple strings (which would otherwise be my first thought). Nor can I reduce the amount of data by applying additional filters to the criteria to the data. So, am wondering if there is another option to using a string (which, to my current understanding, can hold up to about 2 billion characters). I might be able to use a variant but I'd prefer another option if possible. And, that being said, I don't know that a variant would work, either. But I'm open to suggestions. Thanks.
Reason and free inquiry are the only effectual agents against error; they are the natural enemies of error and of error only.

Thomas Jefferson

WAR IS PEACE • FREEDOM IS SLAVERY • IGNORANCE IS STRENGTH

Winston Smith
User avatar
Chopstik
LTD Admin
LTD Admin
LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346
LTD Silver - Rating: 346
 
Posts: 4316
Joined: Thu Oct 11, 2007 2:04 pm
Unrated

Re: Out of string space error

Postby gmmastros on Fri Oct 23, 2009 4:35 pm

Chopstik wrote:we have an application that essentially creates an HTML report


What does the application do with the HTML report? Are you saving it to disk as a .html file? If so, I would encourage you to not let the string build up so big. Occasionally write it to disk.

Also.... really big strings are horrendously slow (in any language). If at all possible, I would encourage you to break this string up in to pieces, possibly storing each piece in an array. Then, when saving the data to disk, just iterate through the array, saving each element as you go.
-George
User avatar
gmmastros
LTD Admin
LTD Admin
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630
 
Posts: 2367
Joined: Tue Oct 09, 2007 5:19 pm
Unrated

Re: Out of string space error

Postby Chopstik on Fri Oct 23, 2009 4:44 pm

gmmastros wrote:
Chopstik wrote:we have an application that essentially creates an HTML report


What does the application do with the HTML report? Are you saving it to disk as a .html file? If so, I would encourage you to not let the string build up so big. Occasionally write it to disk.

Also.... really big strings are horrendously slow (in any language). If at all possible, I would encourage you to break this string up in to pieces, possibly storing each piece in an array. Then, when saving the data to disk, just iterate through the array, saving each element as you go.

The web application will open up the file to be presented to the user in an HTML format. Sorry, wasn't sure exactly how to explain that. Essentially, what happens is that we use a string array and add a new record to the array for each iteration through the recordset. That part appears to process fine. The issue comes at the end when we attempt to join all of the strings in the array together into a single string. It fails with out of string space error.

When you say to iterate through the array saving each element as I go, can you clarify exactly what you mean? Part of the problem is that I need to generate a single string that is then returned to the calling application. Because of other constraints, I am unable to change that setting at this point - though that could potentially change if I have no other option that to re-engineer this piece.
Reason and free inquiry are the only effectual agents against error; they are the natural enemies of error and of error only.

Thomas Jefferson

WAR IS PEACE • FREEDOM IS SLAVERY • IGNORANCE IS STRENGTH

Winston Smith
User avatar
Chopstik
LTD Admin
LTD Admin
LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346
LTD Silver - Rating: 346
 
Posts: 4316
Joined: Thu Oct 11, 2007 2:04 pm
Unrated

Re: Out of string space error

Postby gmmastros on Fri Oct 23, 2009 4:56 pm

What I meant was something like this:

  1. For i = lBound(ArrayName) To uBound(ArrayName)
  2.   ' Save this array element to a file
  3. Next
  4.  


It sounds like that's not going to help you at all (because you need to return this string to the calling application). :(

I think you may need to re-think your process. For example.... take a look at the HTML that is generated. Often times, you can save a lot of space by using CSS files. What I mean is... I've seen stuff where each cell in a table has formatting FOR THAT CELL. Stuff like padding and fonts. Instead, you could create a css file for the HTML document and then remove the formatting from the cells completely.

I guess my advice is.... some how, some way, make the size of the string smaller.
-George
User avatar
gmmastros
LTD Admin
LTD Admin
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630
 
Posts: 2367
Joined: Tue Oct 09, 2007 5:19 pm

Re: Out of string space error

Postby Chopstik on Fri Oct 23, 2009 4:59 pm

Yeah, that's what I am thinking, too. I'm not sure if that is going to be an option but I'm going to see what I can do to see about reducing the size of the string. Problem is, once you get up to ~2 billion characters, just how much can you really cut off to make a difference? And there isn't anything that can truly prevent the problem from re-occurring as it stands right now.

I'll talk with the powers-that-be and explain the options I see at this point and see where we decide to go from here. Thanks!
Reason and free inquiry are the only effectual agents against error; they are the natural enemies of error and of error only.

Thomas Jefferson

WAR IS PEACE • FREEDOM IS SLAVERY • IGNORANCE IS STRENGTH

Winston Smith
User avatar
Chopstik
LTD Admin
LTD Admin
LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346
LTD Silver - Rating: 346
 
Posts: 4316
Joined: Thu Oct 11, 2007 2:04 pm
Unrated

Re: Out of string space error

Postby gmmastros on Fri Oct 23, 2009 5:03 pm

While you're talking to the Head-Rico's about this, suggest to them that returning 2 gigabytes of data to a web browser isn't exactly best practice.
-George
User avatar
gmmastros
LTD Admin
LTD Admin
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630LTD Gold - Rating: 1630
LTD Gold - Rating: 1630LTD Gold - Rating: 1630
 
Posts: 2367
Joined: Tue Oct 09, 2007 5:19 pm

Re: Out of string space error

Postby Chopstik on Fri Oct 23, 2009 5:09 pm

Actually, I have to modify a little bit of what I stated before. It's actually a Delphi app that is calling it and it then takes the returned string value and creates a file and reads the string into the file itself. But the end result is still the same. I need to find another way around returning such a large string file from the VB side. I just don't know what the impact will be over everything else if I make a change like that just yet... So, off to talk with the PTB.
Reason and free inquiry are the only effectual agents against error; they are the natural enemies of error and of error only.

Thomas Jefferson

WAR IS PEACE • FREEDOM IS SLAVERY • IGNORANCE IS STRENGTH

Winston Smith
User avatar
Chopstik
LTD Admin
LTD Admin
LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346
LTD Silver - Rating: 346
 
Posts: 4316
Joined: Thu Oct 11, 2007 2:04 pm
Unrated

Re: Out of string space error

Postby traingamer on Fri Oct 23, 2009 7:43 pm

gmmastros wrote:While you're talking to the Head-Rico's about this, suggest to them that returning 2 gigabytes of data to a web browser isn't exactly best practice.


I've got to question this as well. Not many users want to wait for a 2 gig html file to parse and display in their browsers.
Greg

People demand freedom of speech as a compensation for the freedom of thought which they seldom use. Kierkegaard
User avatar
traingamer
Senior Sage
Senior Sage
LTD Bronze - Rating: 233LTD Bronze - Rating: 233LTD Bronze - Rating: 233LTD Bronze - Rating: 233LTD Bronze - Rating: 233
 
Posts: 1490
Joined: Thu Feb 28, 2008 4:13 pm
Location: St. Louis, MO, US
Unrated

Re: Out of string space error

Postby kaht on Fri Oct 23, 2009 8:08 pm

traingamer wrote:I've got to question this as well. Not many users want to wait for a 2 gig html file to parse and display in their browsers.


Not to mention, it's extremely unlikely that they'll actually read all of it (2 gigs of HTML is a LOT of text, even if you're formatting every single cell of the table). If they're only going to be searching the results of the page for bits of information, why not have them search before the query and only pull out what they need?

I've run into issues like this before with some reporting pages that I've had to develop, and the best solution each time was to break up the string into multiple strings.
#8
AlexCuse is n better than me.
User avatar
kaht
LTD Admin
LTD Admin
LTD Silver - Rating: 252LTD Silver - Rating: 252LTD Silver - Rating: 252LTD Silver - Rating: 252LTD Silver - Rating: 252
LTD Silver - Rating: 252
 
Posts: 1383
Joined: Wed Oct 10, 2007 6:49 pm
Location: STL
Unrated

Re: Out of string space error

Postby Emtucifor on Fri Oct 23, 2009 8:21 pm

Whoa... 2Gb strings? Crazy.

Here's a sort-of-related tip. Writing each next piece of output to a file or to a port is (almost?) always faster than building a string and sending it all at once.

Speed tests will prove this.

  1. Dim BigString
  2. Dim i
  3. For i = 1 to 100000
  4.    BigString = BigString & "i: " & i & "<br />"
  5. Next
  6. Response.Write BigString

  1. Dim BigArray
  2. ReDim BigArray(1 to 100000)
  3. Dim i
  4. For i = 1 to 100000
  5.    BigArray(i) = "i: " & i & "<br />"
  6. Next
  7. Response.Write Join(BigArray)

  1. Dim i
  2. For i = 1 to 100000
  3.    Response.Write "i: "
  4.    Response.Write i
  5.    Response.Write "<br />"
  6. Next
  7. Response.Write BigString

Of these 3, the second will work faster than the first, but the 3rd will be the fastest.

Much of the reason for this is that string concatenation has a hidden Schlemiel the Painter's Algorithm in it.
God cries a little bit every time someone builds a database.
User avatar
Emtucifor
Guru
Guru
LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033
LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033LTD Gold - Rating: 1033
LTD Gold - Rating: 1033
 
Posts: 2835
Joined: Fri May 30, 2008 9:30 pm
Location: Bellingham, WA
Unrated

Re: Out of string space error

Postby dogtacocatgod on Sat Oct 24, 2009 2:32 am

Is is possible that this really is a memory error and there simply was not enough memory left to hold the expanded string?
User avatar
dogtacocatgod
Apprentice
Apprentice
LTD Bronze - Rating: 10
 
Posts: 32
Joined: Thu Aug 13, 2009 1:54 am
Unrated

Re: Out of string space error

Postby Chopstik on Sat Oct 24, 2009 8:52 pm

Thanks for the feedback here. After some discussion, it appears that we are going to be re-writing this particular piece of functionality. Previously, it had not been an issue because none of the users had that much data. Now, we've a user with significant amounts of data being passed back and it clearly is not going to work as it currently stands. For the rest of our customers, it is not an issue (and they don't pass back nearly that amount of data).

So, I'm working on re-writing the solution to see what will work better. Obviously, I won't be passing back strings like that... :o
Reason and free inquiry are the only effectual agents against error; they are the natural enemies of error and of error only.

Thomas Jefferson

WAR IS PEACE • FREEDOM IS SLAVERY • IGNORANCE IS STRENGTH

Winston Smith
User avatar
Chopstik
LTD Admin
LTD Admin
LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346LTD Silver - Rating: 346
LTD Silver - Rating: 346
 
Posts: 4316
Joined: Thu Oct 11, 2007 2:04 pm
Unrated