This project has moved. For the latest updates, please go here.

Memory exception parsing 820 to XML

Dec 3, 2015 at 3:41 PM
Getting an error parsing an 820. Looks like it's an out of memory exception. The 820 file is approx. 28 MB. Any ideas/workarounds would be appreciated. Thanks!

Here is the code that does the parsing:
Function Parse820toXML(strEDI820 As String) As String
    Dim fstream = New FileStream(strEDI820, FileMode.Open, FileAccess.Read)
    Dim parser = New X12Parser()
    Dim interchange = parser.ParseMultiple(fstream).First()
    Dim x12Xml = interchange.Serialize()
    Parse820toXML = x12Xml
End Function
I am getting the exception when it attempts to serialize. Here is the exception details:

System.InvalidOperationException was caught
HResult=-2146233079
Message=There was an error generating the XML document.
Source=System.Xml
StackTrace:
   at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
   at System.Xml.Serialization.XmlSerializer.Serialize(Stream stream, Object o, XmlSerializerNamespaces namespaces)
   at System.Xml.Serialization.XmlSerializer.Serialize(Stream stream, Object o)
   at OopFactory.X12.Parsing.Model.Interchange.Serialize(Stream stream)
   at OopFactory.X12.Parsing.Model.Interchange.Serialize(Boolean suppressComments)
   at OopFactory.X12.Parsing.Model.Interchange.Serialize()
   at FW102W01.BCBSTSHOP820.Parse820toXML(String strEDI820) in C:\Users\j71631a\Documents\Visual Studio 2012\Projects\BCBST SHOP 820\BCBST SHOP 820\BCBSTSHOP820.vb:line 125
   at FW102W01.BCBSTSHOP820.MainProcess() in C:\Users\j71631a\Documents\Visual Studio 2012\Projects\BCBST SHOP 820\BCBST SHOP 820\BCBSTSHOP820.vb:line 73
InnerException: System.OutOfMemoryException
   HResult=-2147024882
   Message=Exception of type 'System.OutOfMemoryException' was thrown.
   Source=mscorlib
   StackTrace:
        at System.IO.MemoryStream.set_Capacity(Int32 value)
        at System.IO.MemoryStream.EnsureCapacity(Int32 value)
        at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
        at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
        at System.IO.StreamWriter.Write(String value)
        at System.Xml.XmlTextEncoder.WriteRawWithSurrogateChecking(String text)
        at System.Xml.XmlTextWriter.WriteComment(String text)
        at OopFactory.X12.Parsing.Model.Segment.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.HierarchicalLoopContainer.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.Loop.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.HierarchicalLoopContainer.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.Loop.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.HierarchicalLoopContainer.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.Transaction.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.FunctionGroup.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.Interchange.WriteXml(XmlWriter writer)
        at OopFactory.X12.Parsing.Model.Segment.System.Xml.Serialization.IXmlSerializable.WriteXml(XmlWriter writer)
        at System.Xml.Serialization.XmlSerializationWriter.WriteSerializable(IXmlSerializable serializable, String name, String ns, Boolean isNullable, Boolean wrapped)
        at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterInterchange.Write1_Interchange(Object o)
   InnerException: 
Coordinator
Dec 3, 2015 at 3:55 PM
Edited Dec 3, 2015 at 3:56 PM
You don't possible work in Oregon do you? You don't have to answer but it sounds like some very large adjustment files from a source in Oregon is producing 820s that their partners can't parse.

Anyways, this is a known limitation of the in memory parsing of x12 using XML and XSLT.

The quick work around is to try and manually split the file. This can be somewhat tedious and error prone if you don't know how to look at the file. There is a command line routine called UnbundleX12 documented here https://x12parser.codeplex.com/documentation for making your files smaller, but I would just use that as a work-around so you can respond to production issues. Try splitting at the transaction set level using ST or higher instead at a loop level.

The desired solution is to switch to a streamed approach of parsing the file. This isn't supported with the xml parser because of the way the xslt has to run on the xml in memory. If you were to instead use the ImportX12 command and load into a database than this will stream the parse into the db, making size a non-issue.

This does require a re-write of everything you have done as xslt into sql queries instead, but is the only scalable solution.

If this is a one of production issue than the UnbundleX12 routine is probably a safer bet.
Dec 3, 2015 at 4:04 PM
Edited Dec 3, 2015 at 4:05 PM
dstrubhar wrote:
You don't possible work in Oregon do you? You don't have to answer but it sounds like some very large adjustment files from a source in Oregon is producing 820s that their partners can't parse.

Anyways, this is a known limitation of the in memory parsing of x12 using XML and XSLT.

The quick work around is to try and manually split the file. This can be somewhat tedious and error prone if you don't know how to look at the file. There is a command line routine called UnbundleX12 documented here https://x12parser.codeplex.com/documentation for making your files smaller, but I would just use that as a work-around so you can respond to production issues. Try splitting at the transaction set level using ST or higher instead at a loop level.

The desired solution is to switch to a streamed approach of parsing the file. This isn't supported with the xml parser because of the way the xslt has to run on the xml in memory. If you were to instead use the ImportX12 command and load into a database than this will stream the parse into the db, making size a non-issue.

This does require a re-write of everything you have done as xslt into sql queries instead, but is the only scalable solution.

If this is a one of production issue than the UnbundleX12 routine is probably a safer bet.
No, I don't work in Oregon. This 820 file is coming from the federal government(Exchange related). This is our first test file we've gotten. I'm hoping not to have to do a complete rewrite.
Coordinator
Dec 3, 2015 at 4:08 PM
Edited Dec 3, 2015 at 4:08 PM
Okay, if your sticking with the xml, than it is best to use the code that is in the UnbundleX12 console to split the file. This will allow you to continue to use the xslt but have it so that it loads smaller file sizes.
Also if you are testing this on your desktop, it might be better to test on the server that is configured similar to your production server so that you know if the memory issue will be a problem there.
Dec 3, 2015 at 6:32 PM
Edited Dec 3, 2015 at 6:33 PM
What about the possibility of unbundling by loop id, serializing each unbundled segment one at a time and running the xml through the xslt and putting out to a csv in append mode? Could it be handled that way? Do you know what the loop id is that you would want to split an 820 on? It's at the membership level, I'm pretty sure. Thanks for your input!
Coordinator
Dec 3, 2015 at 8:01 PM
Using the imbedded spec here https://x12parser.codeplex.com/SourceControl/latest#trunk/src/OopFactory.X12/Specifications/Ansi-820-4010Specification.xml then you would specify "ENT" to the UnbundleX12 routine. The actual transactions are usually an adjustment "ADJ", or a Remittance Advice "RMR", but getting down to "ENT" would bundle all the transactions related to a person down to one file.

This will work but may create a lot of small files, if you are okay with that than that will solve your out of memory issue. Those could be transient files since you just need them to do the parse and you would always keep your original somewhere else. All the control numbers in the headers would stay the same, the only thing that would change between the small files and the original file is that the trailing segments that indicate how me segments in each transaction would obviously change since your small files don't have all the original segments in each broken down file.
Dec 3, 2015 at 8:15 PM
Here is my XSLT where I get the detail. Can you tell from this at which loop I need to unbundle?

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxsl="urn:schemas-microsoft-com:xslt" exclude-result-prefixes="msxsl"
<xsl:output method="text" indent="yes"/>
<xsl:template match="Interchange">
<xsl:apply-templates select="FunctionGroup/Transaction"/>
</xsl:template>

<xsl:template match="Transaction" >
<xsl:variable name="trans" select="."/>
<xsl:variable name="payee" select ="./Loop[@LoopId='N1']/N1[N101='PE']"/>
<xsl:variable name="payor" select ="./Loop[@LoopId='N1']/N1[N101='RM']"/>
<xsl:value-of select="$trans/TRN/TRN02"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR16"/>
<xsl:value-of select="','"/>
<xsl:value-of select="translate($payee/N102,',','')"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payee/N104"/>
<xsl:value-of select="','"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payor/N102"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payor/N104"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payor/../PER/PER02"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payor/../PER/PER04"/>
<xsl:value-of select="','"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$payor/../PER/PER06"/>
<xsl:value-of select="','"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR02"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR03"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR04"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR12"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR13"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR14"/>
<xsl:value-of select="','"/>
<xsl:value-of select="$trans/BPR/BPR15"/>
<xsl:text>&#x0A;</xsl:text>
</xsl:template>
</xsl:stylesheet>