Need to build an interface for XML to SQL Datadata

[Jeff Reed]

sorry copied the wrong ones, I am using the:
[DllImportAttribute(“XMLPlus.dll”, EntryPoint = “XMLPlus_GetItemDataByNaaccrNum“)]
public static extern int XMLPlus_GetItemDataByNaaccrNum(int XmlId,
int naaccrNum);

[DllImportAttribute(“XMLPlus.dll”, EntryPoint = “XMLPlus_GetItemDataByNaaccrId“)]
public static extern int XMLPlus_GetItemDataByNaaccrId(int XmlId,
[InAttribute()] [MarshalAsAttribute(UnmanagedType.LPStr)] string naaccrId);

When I was using the ByNaaccrNum I could refference the string in the callback:
iRtn = XMLPlus_Interop.XMLPlus_GetItemDataByNaaccrNum(xmlId, t_num);
sPPatient = sPPatient + readitem_callback.ToString() + “²”;

The ByNaccrId call is different in that I need to use the pointer that is returned. The process of building a char array and a stringbuilder routine to build the value is what I was hoping there was some examples of since I am having trouble getting the pointer to the value working.

[Kathleen Beaumont]

The callback parameters are the same regardless of whether you are processing by naaccrNum or naaccrId and have the same format. Maybe I’m wrong here, but when I look at this function from my help file, isn’t parameter “value” a C# string? Or are you having to do something to it when you use this code by naaccrNum?

private static void ReadItemByEitherNaaccrIdOrNaaccrNum(
    System.IntPtr owner,
    int patint_ordinal,
    int tumor_ordinal,
    [InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr) string NaaccrId,
    int NaaccrNum,
    [InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr) string value)
{
    // etc.
}

The ByNaccrId call is different in that I need to use the pointer that is returned. The process of building a char array and a stringbuilder routine to build the value is what I was hoping there was some examples of since I am having trouble getting the pointer to the value working.

I interpret the above syntax to be an instruction to C# that the parameter will arrive as a pointer to array of char, which C# converts on the spot to a C# string. You shouldn’t have to do anything else special to be able to use these strings in your code.

May I see the code for the callback function in your code that works when you are retrieving data by value?

Kathleen, still hoping a C#-to-C maven will join this thread!

[Jeff Reed]

This C# code I use works:
iRtn = XMLPlus_Interop.XMLPlus_GetItemDataByNaaccrNum(xmlId, t_num);
sPPatient = sPPatient + readitem_callback.ToString() + “²”;

This C# code the string and returns the pointer value which I need to use to build the string value:
iRtn = XMLPlus_Interop.XMLPlus_GetItemDataByNaaccrId(xmlId, t_id);
sPPatient = sPPatient + readitem_callback.ToString()
// need to fill string value from returned pointer, function requires void def. not char for pointer;

I’ll keep looking, The code works for num may have to go back to that and ask Fabian if he could add naaccrnum to test file

[Kathleen Beaumont]

In your code, where and how is readitem_callback declared?

Is its value assigned in the callback function you passed to XMLPlus_OpenXmlDataFile? That is the function I want to see. From the context of what you wrote above, I’m guessing readitem_callback is a global variable. Its scope will be important to this discussion.

I don’t mean to be haranguing you, it’s just that resolving this issue in this public forum could benefit others. I don’t want to leave the questions unanswered or incomplete.

Thanks, Jeff.

Kathleen

[Jeff Reed]

Kathleen thank you for your patience,

Took a break to lure a large mouth bass but I can report I got the pointer working for the XMLPlus_GetItemDataByNaaccrId function. Somewhere in the code addition to support both the naaccrid and naaccrnum return values I over engineered the pointer assignment, nothing like a full re-write to flush out the bugs. The Edit50.dll worked like a charm when fed a proper pointer.

Steps to success:
Added a Pointer:
XMLPlus_Callbacks.Callback_ReadXmlData funcPtr1 =
new XMLPlus_Callbacks.Callback_ReadXmlData(ReadItemByNaaccrNum);

XMLPlus_Callbacks.Callback_ReadProgress funcPtr2 =
new XMLPlus_Callbacks.Callback_ReadProgress(ProgressFunc);

XMLPlus_Callbacks.Callback_ReadXmlData funcPtr3 =
new XMLPlus_Callbacks.Callback_ReadXmlData(ReadItemByNaaccrId);

Used a switch to set pointer:
if (cb_m1_byname.IsChecked == true)
{ readitem_callback = Marshal.GetFunctionPointerForDelegate(funcPtr3); }
else
{readitem_callback = Marshal.GetFunctionPointerForDelegate(funcPtr1); }

call the function with a switch:
if (cb_m1_byname.IsChecked == true)
{iRtn = XMLPlus_Interop.XMLPlus_GetItemDataByNaaccrId(xmlId, t_id); }
Else
{iRtn = XMLPlus_Interop.XMLPlus_GetItemDataByNaaccrNum(xmlId, t_num); }

I did end up using a global variable to pass the return value but will change that to have the functions call a generic function to build the output.

private static void ReadItemByNaaccrId(
System.IntPtr owner,
int patient_ordinal,
int tumor_ordinal,
[InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr)] string NaaccrId,
int NaaccrNum,[InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr)] string value)
{
globalVariable.sValue = value;
}

The test used a test file generated from the github test file generator with 5000 tumor records. The program loops through a patient with 7 field value calls to the get value function and loops through tumors with 127 calls to the function to build a delimited flat file of tumor with redundant patient info. It took ~11Min to process the file with Debugging on.

[Jeff Reed]

Sorry to report I probably will not be able to attend tomorrow’s meeting.

Attached are some screen shots of the current progress using the XMLPluss.DLL. 100 records .4 milliseconds, 250 records 2.9 Seconds, 500 records ~12sec.

There is a problem with memory not being cleaned out as the application slows down significantly over 500 records. The memory usage builds up till it takes over 14 Min for 5000 records or just fails with memory unavailable. (Box is has 16gb memory, 8gig free). Open to ideas where to look for clearing memory from our seasoned XMLPlus users …

Attachments:

You must be logged in to view attached files.

[Kathleen Beaumont]

There is a problem with memory not being cleaned out

You need to run a profiler to pin down exactly where your trouble is, but just guessing it seems you are allocating memory for repetitive processing and counting on the C# garbage collection process to take care of things. Even as a C++ programmer I learned I have to “think in C” when writing batch processing software. (I love the Standard Template Library, but it’s not the right answer when nanoseconds count.)

So look for ways to re-use allocated objects. For instance, in the C++ code sample for running edits, the XMLPlus API shows that I build a list of allocated objects describing each of the data items I’ll be processing, building it just once, outside the processing loop. Most of the properties of those objects are unchanging (name, length, ParentXMLElement, etc.), but for running edits the value property needs to change for each case. So for each data record, I simply update the data value. Try running edits on large data files (I had a test file of more than 83,000 cases… and boy, did that make a huge XML file! … performance did not degrade from the first case to the last).

Anyway, I recommend using a good profiler. I always relied upon AQTime from SmartBear. I used it for C++ and Object Pascal projects, but it also supports the .NET languages.

[Jeff Reed]

Don’t know what I would do without you KB, You zeroed in on the problem. The garbage collection of C# (PC Version: Recycled Resources), did clean out memory but not the large file content I was building in memory. I changed it to write to file every tumor instead and the 5,000 case test file built in 3.3 seconds down from 14 Min.

Next step integrate with a .Bat file load integrator (oracle and SQL Server).

[Author]

Posts