Tuesday, July 29, 2008

Getting file and line numbers without deploying the PDB files

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/getting_file_and_line_numbers_without_deploying_the_pdb_file.htm]

 

Outline:

  • Problem
  • Inadequate Solutions
  • A Better Way
  • Step 1: Create the pdb2xml Database
  • Step 2: Query the pdb2xml Database Using the IL Offset
  • Step 3: Have Your Application Log the IL Offset
  • Download the source code and demo
  • Conclusion

----

Problem

Enterprise applications will inevitably throw exceptions. The ideal thing to do with these exceptions is to log them, send the results back to the appropriate developer, and then have that developer fix the code. Part of the problem is that the production (i.e. release-mode) logs are often missing helpful information that developers take for granted in debug mode, like the source file and line number.

Inadequate Solutions

One approach is to just settle and not get the extra info. For easy bugs it is sufficient to just have the callstack (from calling the exception’s ToString method) and some extra arbitrary parameters from a custom logger. But what about the bugs that aren't easy?

Another approach is to just dump the PDB files into your production environment. If you put the PDB (program database) files right next to the corresponding DLLs, then .Net will automatically generate the file and line numbers for you in every exception callstack. Recall that the PDB files contain information to reverse-engineer your code, such that a debugger could step through it. So you almost never want to give those files out to the public. That means this approach only works for applications you host, such as an ASP.Net app. But even still, it could be a security risk, or your IT department or deployment team may have a policy against it.

A Better Way

Looking at these inadequate solutions, it makes us appreciate the ideal solution, which would full the two criteria:

  1. Provide you the PDB info like file name and line number
  2. Not require you to ship or deploy the PDB files.

.Net allows you to do this. The “trick” is to get the “IL Offset” and use that to lookup in the PDB files for the exact info you need. Recall that all .Net code gets compiled to Intermediate Language (IL); therefore the IL Offset is just the line number in the IL code. The PDB maps the IL code to your source code. So, this approach has three main steps:

  1. Create the pdb2xml database, this maps your IL to your source code.
  2. Query the pdb2xml database using the IL Offset.
  3. Have your application log the IL Offset.

This approach fulfills our criteria, so let’s explore it in more detail.

Step 1: Create the pdb2xml Database

The PDB files are not plain text, so they’re hard to work with for most people. However, the MSDN debugging team wrote a free tool that lets you convert the PDB files to XML (special thanks to Mike Stall for helping me better understand Pdb2Xml), such that you can easily look up info in them. You can download the “pdb2xml” converter tool from MSDN: http://www.microsoft.com/downloads/details.aspx?familyid=38449a42-6b7a-4e28-80ce-c55645ab1310&displaylang=en

When running the pdb2xml tool, it creates an xml file like so:

<symbols file="TestLoggerApp.Core.dll">
  <files>
    <file id="1" name="C:\Temp\MyApp.Core\Class1.cs" ... />
    <file id="2" name="C:\Temp\MyApp.Core\Class2.cs" ... />
  files>
  <methods>
    <method name="MyApp.Core.Class3.Start3" token="0x6000002">
      <sequencepoints total="1">
        <entry il_offset="0x4" start_row="17" start_column="7"
          end_row="17" end_column="54" file_ref="1" />
      sequencepoints>
      <locals />
    method>
    …

This lists all the files, classes, and method involved. Each method can be looked up via the unique token. The node provides what we ultimately want – the coveted file and line number. We get the file by using entry.file_ref to lookup in the files section, and we get the line number from the entry.start_row attribute.

In order to get the exact node, we will need to know the specific:

  1. Xml file to lookup at, where each xml file maps to a .Net assembly.
  2. Method, which we can obtain from the token. The method name is just extra info for convenience.
  3. IL Offset, which is stored as a hex value.

Because real application usually have many assemblies, ideally we could just point to a bin directory full of pdb files, and have a tool (like an automated build) dynamically generate all the corresponding xml files. We can write a wrapper for pdb2xml to do this.

The biggest issue when writing such a wrapper tool is that pdb2xml, which uses Reflection to dynamically load the assemblies, will get choked up when loading one assembly which contains a class that inherits a class in a different assembly. The easiest way to solve this is to just copy all the targeted assemblies (that you want to generate your xml files for) to the bin of pdb2xml. You could use the ReflectionOnlyAssemblyResolve event to handle resolution errors, but that will provide other problems because you need a physical file, but the event properties only give you the assembly name. While most of the time they’re the same, it will be one more problem to solve when they’re not.

Pdb2xml should handle a variety of cases – assemblies with strong names, third-party references, compiled in release mode, or files that are even several MB big.

ASP.Net applications are a little trickier. Starting with .Net 2.0, ASP allows another compilation model, where every page can get compiled to its own DLL. The easiest way to collect all these DLLs is to run the aspnet_compiler.exe tool, which outputs all the assemblies (and PDBs) to the web’s bin directory. You can read about the aspnet_compiler here: http://msdn.microsoft.com/en-us/library/ms229863(VS.80).aspx, or its MSBuild task equivalent: http://msdn.microsoft.com/en-us/library/ms164291.aspx.

Note that when using the aspnet_compiler, you need to include the debug ‘-d’ switch in order to generate the PDB files. A sample call (ignoring the line breaks) could look like:

aspnet_compiler.exe
-v /my_virtualDir
-p C:\Projects\MyWeb\
-f
-d
-fixednames C:\Projects\precompiledweb\MyWeb\

For convenience, I’ve attached a sample tool - PdbHelper.Cmd.Test (from the download) which will mass-generate these xml files for you. This solves the first step – converting the pdb files to an xml format that we can then query. You can now put those xml files anywhere you want, such as on a shared developer machine.

Step 2: Query the pdb2xml Database Using the IL Offset

Given an xml data island, we can easily query that data. The only thing we need is a key. In this case, we can have the application’s logger generate an xml snippet which some tool or process can then scan for and use it to lookup in the pdb2xml database. Let’s say that our logger gave us the following xml snippet (we’ll discuss how in the next step):

<ILExceptionData>
  <Point module='TestLoggerApp.Core.dll' classFull='TestLoggerApp.Core.Class2'
    methodName='Start2' methodSignature='Int32 Start2(System.String, Boolean)'
    methodToken='0x6000005' ILOffset='11' />
  ...
ILExceptionData
>

For each line in the stack trace, our this XML snippet contains a node. The node has the attributes needed to lookup in the pdb2xml database. These are the three values that we need:

  • module – the .Net module, which directly maps to an xml file.
  • methodToken – the token, which uniquely identifies the method
  • ILOffset – The line, in IL, that threw the exception. Our logger wrote this as decimal, but we can easily convert it to hex.

These values are just included for convenience:

  • classFull – the full, namespace-qualified, name of the class
  • methodName – the actual method’s name
  • methodSignature – the signature, to help troubleshoot overloaded methods

Given this xml snippet, we can have any tool or process consume it. In this case, I wrote a sample WinForm app (PdbHelper.Gui, from the Download) that takes a directory to the pdb2xml database, as well as the Xml Snippet, and performs the lookup. The logic is straightforward, perhaps the only catch is the IL Offset is not always exact; therefore if there is no exact match in the pdb2xml file, round down – i.e. find the previous entry node.

So, the developer could run this app, and it returns the file, line, and column.



While this is a manual GUI app, the logic could be automate for a console app, or other process.

Step 3: Have Your Application Log the IL Offset

The last step is to generate the Xml snippet. Given any Exception, you can use the System.Diagnostics.StackTrace object to determine the IL Offset, method, and module. You first need to create a new StackTrace object using the current Exception. You can then cycle through each StackFrame, getting the relevant data. This logic could be abstracted to its own assembly such that you could easily re-use it across all your applications.

    public static string CreateXmlLog(Exception ex)
    {
      try
      {
        //Get offset:
        System.Diagnostics.StackTrace st = new System.Diagnostics.StackTrace(ex, true);
        System.Diagnostics.StackFrame[] asf = st.GetFrames();

        StringBuilder sb = new StringBuilder();
        sb.Append("\r\n");

        int[] aint = new int[asf.Length];
        for (int i = 0; i < aint.Length; i++)
        {
          System.Diagnostics.StackFrame sf = asf[i];
          sb.Append(string.Format(\r\n",
            sf.GetMethod().Name, sf.GetILOffset(), sf.GetMethod().Module, sf.GetMethod().ReflectedType.FullName,
            sf.GetMethod().ToString(), GetILHexLookup(sf.GetMethod().MetadataToken)));
        }

        sb.Append("\r\n");

        return sb.ToString();
      }
      catch (Exception ex2)
      {
        return "Error in creating ILExceptionData: " + ex2.ToString();
      }
    }
    private static string GetILHexLookup(int intILOffsetDec)
    {
      return "0x" + intILOffsetDec.ToString("X").ToLower();
    }

Download the source code and demo

You can download the complete source code, and an automated demo here.

The package has the following folders:

  • BuildScripts - automated scripts to run everything. This is useful if you wan to integrate the PdbHelper into your own processes.
  • mdbg - the pdb2xml application, with the compiled binaries. This was downloaded from MSDN.
  • PdbHelper.Cmd.Test - the command line tool to create all the xml files from pdb (this wraps the MSDN pdb2xml code)
  • PdbHelper.Core - reusable logic that the command line and GUI both use.
  • PdbHelper.Gui - the windows GUI to easily look up debugging info in the pdb-generated xml files.
  • PdbHelper.Logger - a reusable logger component that takes in an Exception and returns an xml snippet containing the IL offset.
  • TestLoggerApp - a test application to demonstrate all this.

There's not much code to all of this, so you could just reverse engineer it all. But to make it easy, go to the BuildScripts folder and you're see 4 bat files, numbered in order:

  • 0_DeleteBins.bat - cleans up things to "reset" everything (delete bin and obj folders). This is optional, but useful when developing.
  • 1_CompileFramework.bat - compile the PdbHelper framework (you could just open the solution in VS)
  • 2_RunTestApp.bat - runs the test console app, whose whole purpose is to throw an exception, display the IL offset xml snippet, and then write it out to a file for easy use.
  • 3_LookupException.bat - Run the windows GUI app. This passes in command line arguments to automatically populate the xml directory and the IL offset snippet generated in the previous step. You just need to click the "Lookup" button, and it should show you the debug info.

Several of these scripts call MSBuild to run certain tasks. Also, by default, this dumps the pdb2xml files in C:\Temp\pdb2xml.

Conclusion

Using these three steps allows an application to log additional info, from which we can then query the pdb files to find the source file and line number. This extra info can be very useful when debugging, helping to reduce the total cost of ownership.

68 comments:

  1. Sorry to comment on a old post, but I'm getting a 403 Forbidden while trying to download the PDBHelper sample tool.
    Could you make it available again?

    ReplyDelete
    Replies
    1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a .Net developer learn from .Net Core Training in Chennai. or learn thru .Net Core Training in Chennai. Nowadays Dot Net has tons of job opportunities on various vertical industry.
      or Es6 Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry.

      Delete
    2. IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes. IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble.Final Year Projects for CSE

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining .

      Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

      The Angular Training covers a wide range of topics including Angular Directives, Angular Services, and Angular programmability.Angular Training

      Delete
  2. Thanks for your interest. I may have lots the attachement (and pictures) when migrating my site to a new domain a year ago. Sorry for the inconvenience.

    ReplyDelete
  3. Thank you for this article. Very instructive.

    ReplyDelete
  4. This article is very educational. I'm sad that the sample is gone, but thank you.

    ReplyDelete
  5. The supported way to do what you were trying to achieve is to use MiniDumpWriteDump (http://msdn.microsoft.com/en-us/library/windows/desktop/ms680360.aspx). It produces a mini dump you can open in any debugger (VS, WinDbg, etc.) and gives you all the information you request. No tricks required.

    ReplyDelete
  6. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing..
    Believe me I did wrote an post about tutorials for beginners with reference of your blog. 




    Selenium training in bangalore
    Selenium training in Chennai
    Selenium training in Bangalore
    Selenium training in Pune
    Selenium Online training

    ReplyDelete
  7. Great post!
    Thanks for sharing this list!
    It helps me a lot finding a relevant blog in my niche!
    Java Training in Chennai
    Java Training in Coimbatore
    Java Training in Bangalore

    ReplyDelete
  8. thanks for your details it's very use for my works web design company in velachery

    ReplyDelete
  9. Excellent blog thanks for sharing Take care of all your search engine optimization SEO, graphic design, logo creation, social media marketing and digital branding need at one stop - Adhuntt Media. Customer satisfaction and service is our priority - We tread that fine line between projecting your visions into tangible reality! Why wait when you can begin your digital marketing journey with us right now at Adhuntt Media.
    digital marketing company in chennai

    ReplyDelete
  10. Nice blog thanks for sharing Try out different styles and bring your backyard back to life with the best garden service in Chennai - Karuna Nursery Gardens. Right from landscaping, terrace gardening and corporate services and renting plants, we do it all.
    plant nursery in chennai

    ReplyDelete
  11. Excellent blog thanks for sharing Pixies beauty Shop is the best place to buy cosmetics in Chennai. With thousands of premium imported brands to choose from, you’ll never run out of lipstick again. And don’t forget about the best offers and value they provide.
    Cosmetics Shop in Chennai

    ReplyDelete
  12. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    eTechno Soft Solutions is a leading training institute for all kind of the Oracle Training in Bangalore with real-time experienced trainers with 100% Placement Assistance.

    ReplyDelete
  13. Having read this I believed it was extremely technology informative. I appreciate you finding the time and energy to put this content together. I once again find myself personally spending a significant amount of time both reading and leaving comments. But so what, it was still worth it!

    ReplyDelete
  14. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.




    Dot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery






    ReplyDelete
  15. When introducing another application or game, you get a rundown of authorizations that you award the product.먹튀

    ReplyDelete
  16. Building an Access information base typically begins with chipping away at the tables first and getting the right sorts of informational index in your application from the beginning saving you a ton of time and cerebral pains later when computing and alluding to numeric qualities somewhere else.visit website

    ReplyDelete

  17. Thank you .very interesting to read
    https://www.blogger.com/comment.g?blogID=4370421825573178711&postID=6128219949629439185&page=2&token=1615013177957

    ReplyDelete
  18. The writer has outdone himself this time. It is not at all enough; the website is also utmost perfect. I will never forget to visit your site again and again. Convert pdf to word

    ReplyDelete
  19. You have performed a great job on this article. It’s very precise and highly qualitative. You have even managed to make it readable and easy to read. You have some real writing talent. Thank you so much. convert pdf to png

    ReplyDelete
  20. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. video upload

    ReplyDelete
  21. Advances in interchanges and innovation have carried with it an off-shoot marvel that has significantly affected the manner in which people and organizations interface and manage one another. pefile

    ReplyDelete
  22. Advances in interchanges and innovation have carried with it an off-shoot marvel that has significantly affected the manner in which people and organizations interface and manage one another.pe file header

    ReplyDelete
  23. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. 먹튀사이트

    ReplyDelete
  24. Hi there! Nice stuff, do keep me posted when you post again something like this! 먹튀검증

    ReplyDelete
  25. An fascinating discussion is value comment. I think that it is best to write extra on this matter, it won’t be a taboo topic however generally people are not enough to talk on such topics. To the next. Cheers แทงบอล

    ReplyDelete
  26. Without fail, your writing style is top professional; even your website also looks amazing thank you for posting. 스포츠토토

    ReplyDelete
  27. i am always looking for some free stuffs over the internet. there are also some companies which gives free samples. แทงบอล

    ReplyDelete
  28. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. esa letter

    ReplyDelete
  29. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. 토토사이트

    ReplyDelete
  30. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. 메이저사이트

    ReplyDelete
  31. Someone Sometimes with visits your blog regularly and recommended it in my experience to read as well. The way of writing is excellent and also the content is top-notch. Thanks for that insight you provide the readers! cheapest web hosting indiaonohosting

    ReplyDelete
  32. i am always looking for some free stuffs over the internet. there are also some companies which gives free samples. 카지노사이트검증

    ReplyDelete
  33. I really like your writing style, great information, thankyou for posting. 토토사이트

    ReplyDelete
  34. Hi there! Nice post! Please tell us when I will see a follow up! 먹튀검증

    ReplyDelete
  35. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. click here

    ReplyDelete
  36. Efficiently written information. It will be profitable to anybody who utilizes it, counting me. Keep up the good work. For certain I will review out more posts day in and day out. Watergardens at Canberra Showflat

    ReplyDelete
  37. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. apkpure

    ReplyDelete
  38. Keep up the good work , I read few posts on this web site and I conceive that your blog is very interesting and has sets of fantastic information. 토토커뮤니티

    ReplyDelete
  39. i am always looking for some free stuffs over the internet. there are also some companies which gives free samples. 먹튀검증

    ReplyDelete
  40. You delivered such an impressive piece to read, giving every subject enlightenment for us to gain information. Thanks for sharing such information with us due to which my several concepts have been cleared. source

    ReplyDelete
  41. It is truly a well-researched content and excellent wording. I got so engaged in this material that I couldn’t wait reading. I am impressed with your work and skill. Thanks. BizOp

    ReplyDelete
  42. Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. poker88 asia

    ReplyDelete
  43. Every weekend i used to pay a quick visit this web site, because i want enjoyment, for the reason that this this
    web page conations really nice funny data too.

    Feel free to visit my blog - 부산오피

    ReplyDelete
  44. You understand your projects stand out of the crowd. There is something unique about them. It seems to me all of them are brilliant. small capsule lift for home price in india

    ReplyDelete
  45. Thanks for the blog filled with so many information. Stopping by your blog helped me to get what I was looking for. Now my task has become as easy as ABC. 토토사이트

    ReplyDelete