Delphi out of memory как исправить

Есть база данных на SQLite, подключение в проект через FireDAC, компоненты TFDManager, TFDConnection, TFDQuery.
Одна процедура динамически создает 4 таблицы и заполняет данными. На этом этапе все хорошо, объем памяти программы увеличивается на 10 — 12 Мб. Другая процедура многократными запросами к эти четырем таблицам заполняет данными другие 8 таблиц. Здесь начинает расти количество памяти, занимаемой приложением, и в итоге доходит до 2Гб и ошибка out of memory. Пробовал удалять и создавать TFDConnection и TFDQuery под каждый блок данных, но ничего не помогает. На сегодня код такой:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

...
      BasebcConnectionSQL.Connected := false;
      BasebcConnectionSQL.Free;
      BasebcConnectionSQL:= TFDConnection.Create(Self);
      BasebcConnectionSQL.DriverName:= 'SQLite';
      BasebcConnectionSQL.Params.Database :=  PutiMyBaseSQL;
      BasebcConnectionSQL.Connected := true;
      BasebcConnectionSQL.Open();
      FDQ := TFDQuery.Create(Self);
      FDQ.Connection := BasebcConnectionSQL;
...
      FDQ.SQL.Text := 'Select *...';
      FDQ.Open;
...
      FDQ.Free;

расчеты и данные в таблицах без ошибок (перепроверено вручную), но данные не удаляются а накапливаются.
Может кто знает как удалять ненужные данные из памяти приложения?

Go Up to Using ILINK32 and ILINK64 on the Command Line

When using the RAD Studio ILINK32 or ILINK64, you may receive the Fatal: Out of memory message. To handle it, you need to identify the Linker heap that is out of memory and increase its size.

To increase the memory size in the needed heap, do the following:

1. Go to Tools > Options > IDE > Compiling and Running.
2. To see which Linker heap caused the problem, select Diagnostic on the Verbosity menu.
3. Try to link again, and when you see the Fatal: Out of memory message, click the Output tab. It contains additional information that identifies which Linker heap was overrun.
4. On the Output tab, find the following line:

Overrun on linker heap:

The heap that you see after the Overrun on linker heap: message is the heap that was overrun.

Values for all heaps are listed below this line. Please see the example below.

• Look at the two hex values following the heap that is overrun.

• The first value shows the size required by the linker when it realized that the heap would be overrun.
• The second value shows the allocated size.

• Compare the first and second values and pick a value for the heap that exceeds the first value.

5. Now, you can increase the size of the needed heap. To do it, go to Project > Options > C++ Linker and in the Linker Heap Settings section, set the new heap size.

• For the 32-bit Windows platform, there are 5 heaps. In most cases, it’s enough to increase 3 of them: Code Heap Size, Data Heap Size, or Info Heap Size. Please see the sample values used in a reported test case:

• Code Heap Size: 0x14000000
• Data Heap Size: 0x0d000000
• Info Heap Size: 0x0B000000
• RO Data Heap Size: 0
• TDS Heap Size: 0

• For the 64-bit Windows platform, there are 14 heaps. In most cases, it’s enough to increase 2 of them: Dwarf pubtypes Heap Size, or Dwarf aranges Heap Size. Please see the sample values used in a reported test case:

• Code Heap Size: 0
• Data Heap Size: 0
• Dwarf abrbrev Heap Size: 0
• Dwarf aranges Heap Size: 0
• Dwarf info Heap Size: 0
• Dwarf line Heap Size: 0
• Dwarf loc Heap Size: 0
• Dwarf macinfo Heap Size: 0
• Dwarf pubtypes Heap Size: 0x02400000
• Dwarf ranges Heap Size: 0
• Dwarf str Heap Size: 0x08000000
• Info Heap Size: 0
• RO Data Heap Size: 0
• TDS Heap Size: 0

6. Try to link again. In case you see the Fatal: Out of memory message again, repeat the procedure because you might need to increase the memory for other heaps too.

Note: Each application requires increasing different heaps and different values.

Command-Line Flags

In a command-line build, the linker will emit a table like the following example:

Turbo Incremental Link64 6.80 Copyright (c) 1997-2017 Embarcadero Technologies,
Inc.
Overrun on linker heap: info
Linker Heaps
------------
info                   0x12271000  0x12000000
code                   0x0487d000  0x08000000
rodata                 0x00159000  0x06000000
data                   0x029bb000  0x08000000
bss                    0x08000000  0x08000000
dwarf_aranges          0x00010000  0x00200000
dwarf_macinfo          0x00010000  0x00200000
dwarf_pubtypes         0x00275000  0x02000000
dwarf_info             0x04e57000  0x06000000
dwarf_abbrev           0x00040000  0x00200000
dwarf_line             0x0124d000  0x02000000
dwarf_str              0x036e0000  0x06000000
dwarf_loc              0x00070000  0x02000000
dwarf_ranges           0x00275000  0x02000000
Fatal: Out of memory

You can change the heap sizes the linker uses via the -GH flag for ilink32 or ilink64. Use -GH<heapname>=size to allocate a larger heap or multiple heaps. For example, to link the release version of the DLL OWLNext library, internal C++Builder developers use -GHinfo=0x13000000.

See Also

• Using ILINK32 and ILINK64 on the Command Line
• Technical Details About ILINK32 and ILINK64
• C++ Linker

Join Tek-Tips^® Today!

Join your peers on the Internet’s largest technical computer professional community.
It’s easy to join and it’s free.

Here’s Why Members Love Tek-Tips Forums:

• Talk To Other Members
• Notification Of Responses To Questions
• Favorite Forums One Click Access
• Keyword Search Of All Posts, And More…

Register now while it’s still free!

Already a member? Close this window and log in.

Join Us             Close

Добрый день, уважаемые пользователи.
Суть проблемы: При запуске процедуры идет работа в файле размеров в …. мегабайт. И после 5 секунд выскакивает окно out of memory. Хотя файлы до 200 мегабайт нормально.
procedure Tfrm_Main.Run;
var afList: TStringDynArray;
i, j, k: integer;
L, rL, fL: TStringList;
NewName, tmp: String;
begin
memo_Log.Lines.Add(Format(‘Начало обработки: %s’, [DateTimeToStr(Now)]));
memo_Log.Lines.Add(‘///’);
memo_Log.Lines.Add(»);
L := TStringList.Create;
for i := 0 to memo_Folders.Lines.Count — 1 do
begin
afList := TDirectory.GetFiles(memo_Folders.Lines[i], ‘*.txt’, SO);
for j := 0 to Length(afList) — 1 do
L.Add(afList[j]);
end;
rL := TStringList.Create;
rL.Text := Trim(memo_List.Text);
k := 0;
fL := TStringList.Create;
for i := 0 to L.Count — 1 do
begin
memo_Log.Lines.Add(Format(‘Обработка файлов %s’, [L.Strings[i]]));
fL.LoadFromFile(L.Strings[i]);
tmp := ExtractFileName(L.Strings[i]);
tmp := Copy(tmp, 1, Pos(‘.’, tmp) — 1);
NewName := Format(‘%s%s.%s’, [ExtractFilePath(L.Strings[i]),
tmp,
FormatDateTime(‘ddmmyy_hhnn’, Now)]);
if rg_Order.ItemIndex = 1 then
for j := 0 to rL.Count — 1 do
rL.Exchange(j, RandomRange(0, rL.Count));
for j := 0 to fL.Count — 1 do
begin
fL.Strings[j] := StringReplace(fL.Strings[j], edt_Word.Text, rL.Strings[k], RF);
inc(k);
if k = rL.Count then k := 0;
end;
fL.SaveToFile(NewName);
memo_Log.Lines.Add(Format(‘Сохранение под именем %s’, [NewName]));
memo_Log.Lines.Add(»);
end;
memo_Log.Lines.Add(‘///’);
memo_Log.Lines.Add(Format(‘Окончание обработки: %s’, [DateTimeToStr(Now)]));
fL.Free;
rL.Free;
L.Free;
btn_Next.Enabled := false;
LoadBMP(btn_Prev, 5);
btn_Prev.Caption := ‘С начала’;
end;

Ответ

Это не утечка. TStringList имеет ограничение на работу с очень большими файлами (учитывая, что размер файла в памяти после загрузки увеличится вдвое). Как раз 5-10 секунд проходит до того момента, как он упирается в него.
Используйте либо классический readln, либо перерабатывайте под себя TFileStream. Вот пример, давно использованный в нашей программе для работы с большими текстами, переработайте его под себя:
type
TTextFileStream = class(TFileStream)
private
FBuffer: string;
public
function Eof: Boolean;
procedure WriteLn(Text: string);
function ReadLn: string;
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;
function RowsCount: Integer;
end;
implementation
{ TTextFileStream }
function TTextFileStream.Eof: Boolean;
begin
Eof := ((Position — Length(FBuffer)) = Size);
end;
function TTextFileStream.ReadLn: string;
const
BufferLength = 10000;
var
NewBuffer: string;
Readed: Integer;
begin
Readed := 1;
while (Pos(#13, FBuffer) = 0) and (Readed > 0) do begin
SetLength(NewBuffer, BufferLength + 2);
Readed := Read(NewBuffer[1], BufferLength);
SetLength(NewBuffer, Readed);
FBuffer := FBuffer + NewBuffer;
end;
if Pos(#13, FBuffer) > 0 then begin
Result := Copy(FBuffer, 1, Pos(#13, FBuffer) — 1);
Delete(FBuffer, 1, Pos(#13, FBuffer) + 1);
end else begin
Result := FBuffer;
FBuffer := »;
end;
end;
function TTextFileStream.RowsCount: Integer;
begin
Result := 1;
FBuffer := »;
Seek(0, soBeginning);
while not Eof do begin
ReadLn;
Inc(Result);
end;
Seek(0, soBeginning);
end;
function TTextFileStream.Seek(const Offset: Int64;
Origin: TSeekOrigin): Int64;
begin
if Origin = soCurrent then
Result := inherited Seek(Offset — Length(FBuffer), Origin)
else Result := inherited Seek(Offset, Origin);
FBuffer := »;
end;
procedure TTextFileStream.WriteLn(Text: string);
const
NewLine = #13#10;
begin
Seek(0, soCurrent);
Write(Text[1], Length(Text));
Write(NewLine, 2);
end;

FILE_FLAG_NO_BUFFERING places special requirements on use of the file handle, that are not compatible with all functionality of Delphi’s THandleStream class. The principal such requirement is that all access is aligned. By which it is meant that the file pointer is always placced on sector boundaries, and all reads and writes are of multiples of the sector size. The specific failure point here is the Size property.

You are reading a file whose size is not an exact multiple of the sector size. The answer you refer to fails with the error you report when presented a file whose size is not an exact multiple of the sector size. Presumably the author of that code was not aware of the issue and by pure chance used a file whose size was an exact multiple of the sector size.

You can confirm all of this by executing the code with an input file whose size is an exact multiple of 4096.

You likely can use THandleStream with such a file handle but you will need to be careful. Obviously you must avoid Size. You have to respect the alignment requirements. You need to be careful when reading at the end of the file because you will need to read an entire sector even if you know that the logical file ends before the end of the sector. That means using Read rather than ReadBuffer.

Frankly, in my opinion, if you must use unbuffered file access then I don’t believe that the stream abstraction is a good fit. I would work directly with the Windows API.

FILE_FLAG_NO_BUFFERING places special requirements on use of the file handle, that are not compatible with all functionality of Delphi’s THandleStream class. The principal such requirement is that all access is aligned. By which it is meant that the file pointer is always placced on sector boundaries, and all reads and writes are of multiples of the sector size. The specific failure point here is the Size property.

You are reading a file whose size is not an exact multiple of the sector size. The answer you refer to fails with the error you report when presented a file whose size is not an exact multiple of the sector size. Presumably the author of that code was not aware of the issue and by pure chance used a file whose size was an exact multiple of the sector size.

You can confirm all of this by executing the code with an input file whose size is an exact multiple of 4096.

You likely can use THandleStream with such a file handle but you will need to be careful. Obviously you must avoid Size. You have to respect the alignment requirements. You need to be careful when reading at the end of the file because you will need to read an entire sector even if you know that the logical file ends before the end of the sector. That means using Read rather than ReadBuffer.

Frankly, in my opinion, if you must use unbuffered file access then I don’t believe that the stream abstraction is a good fit. I would work directly with the Windows API.

Hey there,

I needed a code to act like a bruteforcer so I found one, but the problem is I don’t want to use TStringList, so I replaced it with array of strings and added some modifications.
The problem is when I run the code, after a certain time I get «Out Of Memory» error.

I hope someone can show me where is my error, I use FreeMem and SetLength to free the array of string each iteration.
Code is below..

Regards

function   bruteforce(astring:   string  ;substr:   string  ;  startlen: integer;
endlen: integer): LongInt; 
var   i,n,x: integer;
bi,bc,br:integer;
npw:   string  ;
counter:integer;
bhlplst: array of string;
bcluster : array of string;
bresults : array of string;
label   step1;
begin
bi := 0;
bc := 0;
br := 0;
//hlplst := TStringList.Create;    //The left column
//cluster := TStringList.Create;   //The middle section
//results := TstringList.Create;   //The combinations created
for   x := 1   to   length(substr)   do begin
//hlplst.Add(substr[x]);
SetLength(bhlplst, bi + 1);
bhlplst[bi] := substr[x];
Inc(bi);
end  ;
 
step1:
for   i := 0   to   {hlplst.Count}bi - 1   do begin
for   n := 1   to   length(astring)   do begin
//npw := hlplst.strings[i]+astring[n];
npw := bhlplst[i] +  astring[n];
//cluster.Add(npw);
 
SetLength(bcluster, bc + 1);
 
bcluster[bc] := npw;
 
Inc(bc);
 
 
if   length(npw) >= startlen   then
begin
//results.Add(npw);
//SetLength(bresults,br + 1);
//bresults[br] := npw;
//Inc(br);
Form1.Memo1.Lines.Add(npw);
end;
end  ;
end  ;
//hlplst.clear;
bi := 0;
SetLength(bhlplst, bi);
FreeMem(bhlplst);
//hlplst.addstrings(cluster);
SetLength(bhlplst, bc );
for counter := 0 to bc - 1 do
begin
bhlplst[bi] := bcluster[counter];
Inc(bi);
end;
//cluster.Clear;
bc := 0;
SetLength(bcluster, bc);
FreeMem(bcluster);
if   length(npw) + 1 <= endlen   then     goto   step1;
//hlplst.Clear;
bi := 0;
SetLength(bhlplst,bi);
FreeMem(bhlplst);
result := 0;
end  ;

Open in new window