Project Description

In this release a technique is implemented typical of the database to access the records. A table of the references in memory is used for accessing the objects on memory mapped files (MMF).

 

Introduction

Working with large structures of data as List, ArrayList and Dictonary (kept in memory for speed) I have encountered out of memory exceptions, especially on the systems to 32bit. Then I have used the structures that use the memory mapped files (MMF). Read about Memory Mapped Files and http://mmf.codeplex.com/.
These structures result to be notably slow above all on systems for the DataWarehouse!

Then the purpose of this project is to sensitively improve the access on the MMF.
In this first release I will apply to a typical technique of the database: the table of the indexes or the references.

Such table for speed (read/write) is maintained in memory and the repeated interrogations are avoided on the presence or less of the object on MMF.

The further improvement is given during the removal of the object, since this operation doesn't happen on the MMF but only on the table of the references.

 

 

What's in the project?

IntelliSearch.MemoryMapped - Thread safe generic data types to use in your code
  • Array<T>, List<T> - autogrowing capabilities with new methods: RemoveAt, Remove, Exist and Contains (both Thread safe)!

 

Benchmark on Intel Atom @1.60Ghz, 2GB RAM, HD 4200rpm (Ordered insertion)

Test Time
Insert (Ideal) 00:00:05.2187500
Insert (MMF) 00:00:00.4843750(92x faster)
Contains (Ideal) 00:00:00.0156250
Contains (MMF) 00:00:01.4375000(99x slower)
Fetch (Ideal) 00:00:00.0156250
Fetch (MMF) 00:00:00.4062500(97x slower)
RemoveAt (Ideal) 00:00:00.4375000
RemoveAt (MMF) 00:12:14.5781250(99x slower)

and after the improvement...

Insert (MMF) 00:00:00.5156250(90x faster)
Contains (MMF) 00:00:01.9687500(99x slower)
Fetch (MMF) 00:00:00.2968750(96x slower)
RemoveAt (MMF) 00:00:00.0156250(96x faster)

 

The results of the tests have been performed for the Insert and Fetch on 100000 objects and 1000 on the Contains and RemoveAt.
The Insert and Fetch in MMF result a little slower after the optimization respectively: 2x and 1x slower (overhead).
You have an improvement of speed on the method RemoveAt (96x) after the optimization.

The future

  • Implement more methods (Shrink) on the classes and improvement performances on Fetch and Contains.
  • Improvement class Dictionary

Last edited Oct 29, 2010 at 9:03 AM by Isterop, version 5