Invalid character

May 1, 2013 at 10:25 PM
Some of the files I've been working with have an invalid character (hex 0x1A) appended to the end of the file which causes errors during parsing.
OopFactory.X12.Parsing.Model.TransactionValidationException was unhandled
  Message=Segment '0x1A' in segment position 185 within transaction '31042' cannot be identified within the supplied specification for transaction set 837 in any of the expected loops: 2000A[1],2000B[8],2300,2400,2420A.  To change this to a warning, pass throwExceptionOnSyntaxErrors = false to the X12Parser constructor.
Parameter name: 837
  Source=OopFactory.X12
  ParamName=837
  ControlNumber=31042
  ElementId=""
  TransactionCode=837
  Value=0x1A
  StackTrace:
       at OopFactory.X12.Parsing.X12Parser.ParseMultiple(Stream stream, Encoding encoding) in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\OopFactory.X12\Parsing\X12Parser.cs:line 238
       at X12Viewer.Form1.ParseFile() in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Form1.cs:line 46
       at X12Viewer.Form1.button1_Click(Object sender, EventArgs e) in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Form1.cs:line 27
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
       at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
       at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.Run(Form mainForm)
       at X12Viewer.Program.Main() in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Program.cs:line 18
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 
I tried the throwExceptionOnSyntaxErrors = false suggestion, but I get the following error on the attempt to call OnParserWarning. Also it looks like the transform services and the X12.Hipaa project don't support this setting, correct?
System.NullReferenceException was unhandled
  Message=Object reference not set to an instance of an object.
  Source=OopFactory.X12
  StackTrace:
       at OopFactory.X12.Parsing.X12Parser.ParseMultiple(Stream stream, Encoding encoding) in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\OopFactory.X12\Parsing\X12Parser.cs:line 245
       at X12Viewer.Form1.ParseFile() in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Form1.cs:line 46
       at X12Viewer.Form1.button1_Click(Object sender, EventArgs e) in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Form1.cs:line 27
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
       at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
       at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.Run(Form mainForm)
       at X12Viewer.Program.Main() in C:\DevTFS\Tools\X12Parser\GEMMS-branch\trunk\src\X12Viewer\Program.cs:line 18
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 
I was able to solve this particular issue by changing this line in X12StreamReader.ReadNextSegment()
return sb.ToString().TrimStart();
to this
return sb.ToString().TrimStart().Trim('\x1A');
That works for me, but I didn't know if there would be a better place in the code to strip out the invalid character.

I would appreciate any comments/suggestions.

Thanks
Coordinator
May 2, 2013 at 1:25 AM
I have added changeset 25742 which will allow you to inject a list of chars to ignore and it will ignore them anywhere they exist in the file. This should give you more flexibility if you end up experiencing this again with some other characters. Occasionally there have been unprintable characters like this used as segment delimiters, so this should be configured by the user what characters they want to ignore.

Let me know if this solves your problem.
May 6, 2013 at 5:26 PM
That worked when I'm creating the X12Parser object directly as a test, but I'm primarily using the OopFactory.X12.Hipaa project (and also the HTML transformation).

So this works
            OopFactory.X12.Parsing.X12Parser parser = new OopFactory.X12.Parsing.X12Parser(new SpecificationFinder(), false, new char[] { '\x1A' });
            using (FileStream fs = new FileStream(x12Filename, FileMode.Open, FileAccess.Read))
            {
                var interchanges = parser.ParseMultiple(fs, encoding);
            }
But when trying to create a ClaimDocument, I don't see a way to pass in the invalid characters
            var service = new ClaimFormTransformationService(
                new ProfessionalClaimToHcfa1500FormTransformation("HCFA1500_Red.gif"),
                new InstitutionalClaimToUB04ClaimFormTransformation("UB04_Red.gif"),
                new DentalClaimToJ400FormTransformation("ADAJ400_Red.gif"));

            ClaimDocument claimDoc;

            using (FileStream inputFilestream = new FileStream(x12Filename, FileMode.Open, FileAccess.Read))
            {
                claimDoc = service.Transform837ToClaimDocument(inputFilestream);
            }
Or when converting to HTML for viewing.
            var htmlService = new OopFactory.X12.Transformations.X12HtmlTransformationService(new OopFactory.X12.Transformations.X12EdiParsingService(suppressComments: false));
            Stream ediFile = new FileStream(x12Filename, FileMode.Open, FileAccess.Read);
            string html = htmlService.Transform(new StreamReader(ediFile).ReadToEnd());
            webBrowser1.DocumentText = html;
Thanks!
Coordinator
May 7, 2013 at 10:43 PM
I have checked in the fix in changeset 25950. This will allow you to inject your configuration of the X12Parser into the ClaimTransformationService and ClaimFormTransformationService. It will be in the next release.
May 9, 2013 at 4:22 PM
That works great for me, thanks!

How about changing the X12EdiParsingService class (src\OopFactory.X12\Transformations\X12EdiParsingService.cs) to the code below, so we can pass the X12Parser into that as well? Also looked like the suppressComments was not being pass through for the "public X12EdiParsingService(bool suppressComments, ISpecificationFinder specFinder)" constructor so I added that line as well.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using OopFactory.X12.Parsing;
using OopFactory.X12.Parsing.Model;

namespace OopFactory.X12.Transformations
{
    public class X12EdiParsingService : ITransformationService
    {
        private bool _suppressComments;
        private X12Parser _parser;

        public X12EdiParsingService(bool suppressComments)
        {
            _parser = new X12Parser();
            _suppressComments = suppressComments;
        }

        public X12EdiParsingService(bool suppressComments, ISpecificationFinder specFinder)
        {
            _parser = new X12Parser(specFinder, true);
            _suppressComments = suppressComments;
        }

        public X12EdiParsingService(bool suppressComments, X12Parser parser)
        {
            _parser = parser;
            _suppressComments = suppressComments;
        }

        public string Transform(string x12)
        {
            Interchange interchange = _parser.ParseMultiple(new MemoryStream(Encoding.ASCII.GetBytes(x12))).FirstOrDefault();
            return interchange.Serialize(_suppressComments);
        }

    }
}
May 9, 2013 at 10:08 PM
Also tried an attempt at adding the ignored characters to the X12AcknowledgmentService.
namespace OopFactory.X12.Validation
{
    public class X12AcknowledgmentService
    {
        ISpecificationFinder _specFinder;
        private char[] _ignoredChars;

        public X12AcknowledgmentService(ISpecificationFinder specFinder, char[] ignoredChars)
        {
            _specFinder = specFinder;
            _ignoredChars = ignoredChars;
        }

        public X12AcknowledgmentService(ISpecificationFinder specFinder)
            : this(specFinder, new char[] { })
        {
        }

        public X12AcknowledgmentService()
            : this(new SpecificationFinder(), new char[] { })
        {
        }

        public List<FunctionalGroupResponse> AcknowledgeTransactions(Stream x12Stream)
        {
            return AcknowledgeTransactions(x12Stream, Encoding.UTF8);
        }

        public virtual List<FunctionalGroupResponse> AcknowledgeTransactions(Stream x12Stream, Encoding encoding)
        {
            var responses = new Dictionary<string, FunctionalGroupResponse>();

            using (var reader = new X12StreamReader(x12Stream, encoding, _ignoredChars))
            {
                var trans = reader.ReadNextTransaction();
                while (!string.IsNullOrEmpty(trans.Transactions.First()))
                {
Coordinator
May 12, 2013 at 2:58 PM
This is fixed and release in version 3.0.5.