Unbundle X12 function results in "Thread was being aborted" error

Jun 20 at 3:24 PM
Dear Dstrubhar.

We have been using the X12 Parser to Unbundle and Parse 837 and 835s for couple years now without any issues. Thank you again for to you and the entire Team for this great product and your continued support.

We are running into a problem with the unbundling function for a set of 4010 Institutional Claims. For some reason, the unbundle function just hangs our process (running in a windows service). There is no error at that point. Its only after we stop the Service, I see this error: "Thread was being aborted". The code reaches the catch block only when after onStop( ) function of the Windows Service gets executed. Up until that point, the X12 parser doesn't respond. I also dont see any files in the unbundled directory.

I don't have any logs I can provide. How would you recommend I troubleshoot this issue?

Your help, as always, is greatly appreciated..
Coordinator
Jun 21 at 4:03 PM
Thiis is usually due to the size of the file being too large and the server not having enough memory to pull the whole Interchange object into memory. Usually though when that is the problem users will report that they get a OutOfMemoryException.

Please let me know the size of the largest file that you successfully processed on this server and let me know the size of the file that is failing.

If the size is the problem, then you may trying a two step unbundle where you unbundle by the transaction or the provider loops first and then unbundle by the claim loop. Another option would be to use the maxBatchSize setting that is available in the console app's config settings. This will limit the amount of the file that is parsed at one time in case there are memory issues, I'm not sure if that will apply to your case if you are only using it to Unbundle.

The reason the Unbundle can be very memory intensive is that you are taking an normalized x12 file and repeating all the header, and provider information as you unbundle it into individual files.

Since you are running it through a windows server, another possible technique for troubleshooting is to unbundle with the console app directly and see if that will report an exception that might be more useful to troubleshoot.

Though the initial release kept the parsing decoupled from any database, last year I did release a feature that would allow you to parse directly into a sql server database. This is an alternative to unbundling the file directly in that it doesn't require as much memory because it streams everything to the database. Once in a SQL Server database there is a table-returning UDF call GetTransactionSegments that has the same effect as unbundling in that if you send in the Id of any claim loop it will give you all the segments of that claim similar to the unbundle feature. I would consider this a last resort for you since it would require a rewrite of your process, but the technique is there for scalability when the parsing of the file directly can't be done because of file size.
Jun 21 at 6:09 PM
Dear Sir,
Thank you so much for your detailed response. I wasn't aware of the new feature you introduced recently that writes to a SQL server. That's really good to know.

In the current issue, the file size is significantly smaller ~200-300 kb. We have had no issue unbundling claims ~20-30MB in the past using the current release.

When running in the debugger, it hangs when running this loop:

// hangs in this loop
                List<Interchange> list = new List<Interchange>();
                X12StreamReader reader = new X12StreamReader(fs, Encoding.UTF8);
                X12FlatTransaction transaction = reader.ReadNextTransaction();
                while (!string.IsNullOrEmpty(transaction.Transactions.First()))
                {
                    string x12 = transaction.ToString();
                    var interchange = parser.ParseMultiple(x12).First();
                    transaction = reader.ReadNextTransaction();
                }
There's no exception caught until after the Windows Service is stopped. Any suggestions on how we can diagnose the problem?

Thanks a bunch for all your help,
Niitn
Jun 21 at 6:11 PM
There was a typo in the code snippet. I missed a line. Here's the code again:
  .....
  .....
  .....
            OopFactory.X12.Parsing.X12Parser parser = new OopFactory.X12.Parsing.X12Parser();
            string unbundledOutputFilename;
            string fileName = Path.GetFileName(origX12FileName);

            System.IO.Directory.CreateDirectory(unbundleDir);

            fs = new FileStream(origX12FileName, FileMode.Open);
            List<Interchange> list = new List<Interchange>();

                X12StreamReader reader = new X12StreamReader(fs, Encoding.UTF8);
                X12FlatTransaction transaction = reader.ReadNextTransaction();
                while (!string.IsNullOrEmpty(transaction.Transactions.First()))
                {
                    string x12 = transaction.ToString();
                    var interchange = parser.ParseMultiple(x12).First();
                    list.AddRange(parser.UnbundleByLoop(interchange, "2300"));
                    transaction = reader.ReadNextTransaction();
                }

  .....
  .....
  .....
Jun 30 at 7:02 PM
Dear Dstrubhar,
I ran the claim through the debugger and the program just hangs in this function in X12Parser.cs.

public List<Interchange> UnbundleByLoop(Interchange interchange, string loopId)
    {
        char terminator = interchange._delimiters.SegmentTerminator;
        var service = new UnbundlingService(interchange._delimiters.SegmentTerminator);
        string isa = interchange.SegmentString;
        string iea = interchange.TrailerSegments.First().SegmentString;
        List<string> list = new List<string>();
        foreach (FunctionGroup group in interchange.FunctionGroups)
            foreach (Transaction transaction in group.Transactions)
            {
                service.UnbundleHLoops(list, transaction, loopId);
            }

        List<Interchange> interchanges = new List<Interchange>();
        foreach (var item in list)
        {
            StringBuilder x12 = new StringBuilder();
            x12.AppendFormat("{0}{1}", isa, terminator);
            x12.Append(item);
            x12.AppendFormat("{0}{1}", iea, terminator);
            using (MemoryStream mstream = new MemoryStream(Encoding.ASCII.GetBytes(x12.ToString())))
            {
                interchanges.AddRange(ParseMultiple(mstream));
            }
        }
        return interchanges;
    }
Some of the failed claims don't appear to have the ~IEA1000277599~ text. They seem to end abruptly. Do you think that could cause the Unbundle routine to just hang? Or fall into an infinite loop? Would really appreciate any help.

Thanks in advance for all your help.
nk5491
Coordinator
Jul 1 at 10:55 PM
I would need to either get an exception message or a test file to reproduce the error.

If you are able to de-identify your file with a tool such as the one here: (http://etasoft.com/ot.htm), then I can try to reproduce your problem.

I might be able to add a tool that will de-identify everything but the structure of your X12 message, but it might be a few more weeks before I can produce that.