Tuesday, July 29, 2008

Getting file and line numbers without deploying the PDB files

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/getting_file_and_line_numbers_without_deploying_the_pdb_file.htm]

 

Outline:

  • Problem
  • Inadequate Solutions
  • A Better Way
  • Step 1: Create the pdb2xml Database
  • Step 2: Query the pdb2xml Database Using the IL Offset
  • Step 3: Have Your Application Log the IL Offset
  • Download the source code and demo
  • Conclusion

----

Problem

Enterprise applications will inevitably throw exceptions. The ideal thing to do with these exceptions is to log them, send the results back to the appropriate developer, and then have that developer fix the code. Part of the problem is that the production (i.e. release-mode) logs are often missing helpful information that developers take for granted in debug mode, like the source file and line number.

Inadequate Solutions

One approach is to just settle and not get the extra info. For easy bugs it is sufficient to just have the callstack (from calling the exception’s ToString method) and some extra arbitrary parameters from a custom logger. But what about the bugs that aren't easy?

Another approach is to just dump the PDB files into your production environment. If you put the PDB (program database) files right next to the corresponding DLLs, then .Net will automatically generate the file and line numbers for you in every exception callstack. Recall that the PDB files contain information to reverse-engineer your code, such that a debugger could step through it. So you almost never want to give those files out to the public. That means this approach only works for applications you host, such as an ASP.Net app. But even still, it could be a security risk, or your IT department or deployment team may have a policy against it.

A Better Way

Looking at these inadequate solutions, it makes us appreciate the ideal solution, which would full the two criteria:

  1. Provide you the PDB info like file name and line number
  2. Not require you to ship or deploy the PDB files.

.Net allows you to do this. The “trick” is to get the “IL Offset” and use that to lookup in the PDB files for the exact info you need. Recall that all .Net code gets compiled to Intermediate Language (IL); therefore the IL Offset is just the line number in the IL code. The PDB maps the IL code to your source code. So, this approach has three main steps:

  1. Create the pdb2xml database, this maps your IL to your source code.
  2. Query the pdb2xml database using the IL Offset.
  3. Have your application log the IL Offset.

This approach fulfills our criteria, so let’s explore it in more detail.

Step 1: Create the pdb2xml Database

The PDB files are not plain text, so they’re hard to work with for most people. However, the MSDN debugging team wrote a free tool that lets you convert the PDB files to XML (special thanks to Mike Stall for helping me better understand Pdb2Xml), such that you can easily look up info in them. You can download the “pdb2xml” converter tool from MSDN: http://www.microsoft.com/downloads/details.aspx?familyid=38449a42-6b7a-4e28-80ce-c55645ab1310&displaylang=en

When running the pdb2xml tool, it creates an xml file like so:

<symbols file="TestLoggerApp.Core.dll">
  <files>
    <file id="1" name="C:\Temp\MyApp.Core\Class1.cs" ... />
    <file id="2" name="C:\Temp\MyApp.Core\Class2.cs" ... />
  files>
  <methods>
    <method name="MyApp.Core.Class3.Start3" token="0x6000002">
      <sequencepoints total="1">
        <entry il_offset="0x4" start_row="17" start_column="7"
          end_row="17" end_column="54" file_ref="1" />
      sequencepoints>
      <locals />
    method>
    …

This lists all the files, classes, and method involved. Each method can be looked up via the unique token. The node provides what we ultimately want – the coveted file and line number. We get the file by using entry.file_ref to lookup in the files section, and we get the line number from the entry.start_row attribute.

In order to get the exact node, we will need to know the specific:

  1. Xml file to lookup at, where each xml file maps to a .Net assembly.
  2. Method, which we can obtain from the token. The method name is just extra info for convenience.
  3. IL Offset, which is stored as a hex value.

Because real application usually have many assemblies, ideally we could just point to a bin directory full of pdb files, and have a tool (like an automated build) dynamically generate all the corresponding xml files. We can write a wrapper for pdb2xml to do this.

The biggest issue when writing such a wrapper tool is that pdb2xml, which uses Reflection to dynamically load the assemblies, will get choked up when loading one assembly which contains a class that inherits a class in a different assembly. The easiest way to solve this is to just copy all the targeted assemblies (that you want to generate your xml files for) to the bin of pdb2xml. You could use the ReflectionOnlyAssemblyResolve event to handle resolution errors, but that will provide other problems because you need a physical file, but the event properties only give you the assembly name. While most of the time they’re the same, it will be one more problem to solve when they’re not.

Pdb2xml should handle a variety of cases – assemblies with strong names, third-party references, compiled in release mode, or files that are even several MB big.

ASP.Net applications are a little trickier. Starting with .Net 2.0, ASP allows another compilation model, where every page can get compiled to its own DLL. The easiest way to collect all these DLLs is to run the aspnet_compiler.exe tool, which outputs all the assemblies (and PDBs) to the web’s bin directory. You can read about the aspnet_compiler here: http://msdn.microsoft.com/en-us/library/ms229863(VS.80).aspx, or its MSBuild task equivalent: http://msdn.microsoft.com/en-us/library/ms164291.aspx.

Note that when using the aspnet_compiler, you need to include the debug ‘-d’ switch in order to generate the PDB files. A sample call (ignoring the line breaks) could look like:

aspnet_compiler.exe
-v /my_virtualDir
-p C:\Projects\MyWeb\
-f
-d
-fixednames C:\Projects\precompiledweb\MyWeb\

For convenience, I’ve attached a sample tool - PdbHelper.Cmd.Test (from the download) which will mass-generate these xml files for you. This solves the first step – converting the pdb files to an xml format that we can then query. You can now put those xml files anywhere you want, such as on a shared developer machine.

Step 2: Query the pdb2xml Database Using the IL Offset

Given an xml data island, we can easily query that data. The only thing we need is a key. In this case, we can have the application’s logger generate an xml snippet which some tool or process can then scan for and use it to lookup in the pdb2xml database. Let’s say that our logger gave us the following xml snippet (we’ll discuss how in the next step):

<ILExceptionData>
  <Point module='TestLoggerApp.Core.dll' classFull='TestLoggerApp.Core.Class2'
    methodName='Start2' methodSignature='Int32 Start2(System.String, Boolean)'
    methodToken='0x6000005' ILOffset='11' />
  ...
ILExceptionData
>

For each line in the stack trace, our this XML snippet contains a node. The node has the attributes needed to lookup in the pdb2xml database. These are the three values that we need:

  • module – the .Net module, which directly maps to an xml file.
  • methodToken – the token, which uniquely identifies the method
  • ILOffset – The line, in IL, that threw the exception. Our logger wrote this as decimal, but we can easily convert it to hex.

These values are just included for convenience:

  • classFull – the full, namespace-qualified, name of the class
  • methodName – the actual method’s name
  • methodSignature – the signature, to help troubleshoot overloaded methods

Given this xml snippet, we can have any tool or process consume it. In this case, I wrote a sample WinForm app (PdbHelper.Gui, from the Download) that takes a directory to the pdb2xml database, as well as the Xml Snippet, and performs the lookup. The logic is straightforward, perhaps the only catch is the IL Offset is not always exact; therefore if there is no exact match in the pdb2xml file, round down – i.e. find the previous entry node.

So, the developer could run this app, and it returns the file, line, and column.



While this is a manual GUI app, the logic could be automate for a console app, or other process.

Step 3: Have Your Application Log the IL Offset

The last step is to generate the Xml snippet. Given any Exception, you can use the System.Diagnostics.StackTrace object to determine the IL Offset, method, and module. You first need to create a new StackTrace object using the current Exception. You can then cycle through each StackFrame, getting the relevant data. This logic could be abstracted to its own assembly such that you could easily re-use it across all your applications.

    public static string CreateXmlLog(Exception ex)
    {
      try
      {
        //Get offset:
        System.Diagnostics.StackTrace st = new System.Diagnostics.StackTrace(ex, true);
        System.Diagnostics.StackFrame[] asf = st.GetFrames();

        StringBuilder sb = new StringBuilder();
        sb.Append("\r\n");

        int[] aint = new int[asf.Length];
        for (int i = 0; i < aint.Length; i++)
        {
          System.Diagnostics.StackFrame sf = asf[i];
          sb.Append(string.Format(\r\n",
            sf.GetMethod().Name, sf.GetILOffset(), sf.GetMethod().Module, sf.GetMethod().ReflectedType.FullName,
            sf.GetMethod().ToString(), GetILHexLookup(sf.GetMethod().MetadataToken)));
        }

        sb.Append("\r\n");

        return sb.ToString();
      }
      catch (Exception ex2)
      {
        return "Error in creating ILExceptionData: " + ex2.ToString();
      }
    }
    private static string GetILHexLookup(int intILOffsetDec)
    {
      return "0x" + intILOffsetDec.ToString("X").ToLower();
    }

Download the source code and demo

You can download the complete source code, and an automated demo here.

The package has the following folders:

  • BuildScripts - automated scripts to run everything. This is useful if you wan to integrate the PdbHelper into your own processes.
  • mdbg - the pdb2xml application, with the compiled binaries. This was downloaded from MSDN.
  • PdbHelper.Cmd.Test - the command line tool to create all the xml files from pdb (this wraps the MSDN pdb2xml code)
  • PdbHelper.Core - reusable logic that the command line and GUI both use.
  • PdbHelper.Gui - the windows GUI to easily look up debugging info in the pdb-generated xml files.
  • PdbHelper.Logger - a reusable logger component that takes in an Exception and returns an xml snippet containing the IL offset.
  • TestLoggerApp - a test application to demonstrate all this.

There's not much code to all of this, so you could just reverse engineer it all. But to make it easy, go to the BuildScripts folder and you're see 4 bat files, numbered in order:

  • 0_DeleteBins.bat - cleans up things to "reset" everything (delete bin and obj folders). This is optional, but useful when developing.
  • 1_CompileFramework.bat - compile the PdbHelper framework (you could just open the solution in VS)
  • 2_RunTestApp.bat - runs the test console app, whose whole purpose is to throw an exception, display the IL offset xml snippet, and then write it out to a file for easy use.
  • 3_LookupException.bat - Run the windows GUI app. This passes in command line arguments to automatically populate the xml directory and the IL offset snippet generated in the previous step. You just need to click the "Lookup" button, and it should show you the debug info.

Several of these scripts call MSBuild to run certain tasks. Also, by default, this dumps the pdb2xml files in C:\Temp\pdb2xml.

Conclusion

Using these three steps allows an application to log additional info, from which we can then query the pdb files to find the source file and line number. This extra info can be very useful when debugging, helping to reduce the total cost of ownership.

166 comments:

  1. Sorry to comment on a old post, but I'm getting a 403 Forbidden while trying to download the PDBHelper sample tool.
    Could you make it available again?

    ReplyDelete
  2. Thanks for your interest. I may have lots the attachement (and pictures) when migrating my site to a new domain a year ago. Sorry for the inconvenience.

    ReplyDelete
  3. Thank you for this article. Very instructive.

    ReplyDelete
  4. This article is very educational. I'm sad that the sample is gone, but thank you.

    ReplyDelete
  5. The supported way to do what you were trying to achieve is to use MiniDumpWriteDump (http://msdn.microsoft.com/en-us/library/windows/desktop/ms680360.aspx). It produces a mini dump you can open in any debugger (VS, WinDbg, etc.) and gives you all the information you request. No tricks required.

    ReplyDelete
  6. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing..
    Believe me I did wrote an post about tutorials for beginners with reference of your blog. 




    Selenium training in bangalore
    Selenium training in Chennai
    Selenium training in Bangalore
    Selenium training in Pune
    Selenium Online training

    ReplyDelete
  7. Great post!
    Thanks for sharing this list!
    It helps me a lot finding a relevant blog in my niche!
    Java Training in Chennai
    Java Training in Coimbatore
    Java Training in Bangalore

    ReplyDelete
  8. Excellent blog thanks for sharing Take care of all your search engine optimization SEO, graphic design, logo creation, social media marketing and digital branding need at one stop - Adhuntt Media. Customer satisfaction and service is our priority - We tread that fine line between projecting your visions into tangible reality! Why wait when you can begin your digital marketing journey with us right now at Adhuntt Media.
    digital marketing company in chennai

    ReplyDelete
  9. Nice blog thanks for sharing Try out different styles and bring your backyard back to life with the best garden service in Chennai - Karuna Nursery Gardens. Right from landscaping, terrace gardening and corporate services and renting plants, we do it all.
    plant nursery in chennai

    ReplyDelete
  10. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    eTechno Soft Solutions is a leading training institute for all kind of the Oracle Training in Bangalore with real-time experienced trainers with 100% Placement Assistance.

    ReplyDelete
  11. When introducing another application or game, you get a rundown of authorizations that you award the product.먹튀

    ReplyDelete
  12. Building an Access information base typically begins with chipping away at the tables first and getting the right sorts of informational index in your application from the beginning saving you a ton of time and cerebral pains later when computing and alluding to numeric qualities somewhere else.visit website

    ReplyDelete

  13. Thank you .very interesting to read
    https://www.blogger.com/comment.g?blogID=4370421825573178711&postID=6128219949629439185&page=2&token=1615013177957

    ReplyDelete
  14. The writer has outdone himself this time. It is not at all enough; the website is also utmost perfect. I will never forget to visit your site again and again. Convert pdf to word

    ReplyDelete
  15. You have performed a great job on this article. It’s very precise and highly qualitative. You have even managed to make it readable and easy to read. You have some real writing talent. Thank you so much. convert pdf to png

    ReplyDelete
  16. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. video upload

    ReplyDelete
  17. Advances in interchanges and innovation have carried with it an off-shoot marvel that has significantly affected the manner in which people and organizations interface and manage one another. pefile

    ReplyDelete
  18. Advances in interchanges and innovation have carried with it an off-shoot marvel that has significantly affected the manner in which people and organizations interface and manage one another.pe file header

    ReplyDelete
  19. Hi there! Nice stuff, do keep me posted when you post again something like this! 먹튀검증

    ReplyDelete
  20. Without fail, your writing style is top professional; even your website also looks amazing thank you for posting. 스포츠토토

    ReplyDelete
  21. i am always looking for some free stuffs over the internet. there are also some companies which gives free samples. แทงบอล

    ReplyDelete
  22. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. 메이저사이트

    ReplyDelete
  23. Someone Sometimes with visits your blog regularly and recommended it in my experience to read as well. The way of writing is excellent and also the content is top-notch. Thanks for that insight you provide the readers! cheapest web hosting indiaonohosting

    ReplyDelete
  24. Hi there! Nice post! Please tell us when I will see a follow up! 먹튀검증

    ReplyDelete
  25. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. click here

    ReplyDelete
  26. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. apkpure

    ReplyDelete
  27. It is truly a well-researched content and excellent wording. I got so engaged in this material that I couldn’t wait reading. I am impressed with your work and skill. Thanks. BizOp

    ReplyDelete
  28. Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. poker88 asia

    ReplyDelete
  29. Every weekend i used to pay a quick visit this web site, because i want enjoyment, for the reason that this this
    web page conations really nice funny data too.

    Feel free to visit my blog - 부산오피

    ReplyDelete
  30. You understand your projects stand out of the crowd. There is something unique about them. It seems to me all of them are brilliant. small capsule lift for home price in india

    ReplyDelete
  31. Thanks for the blog filled with so many information. Stopping by your blog helped me to get what I was looking for. Now my task has become as easy as ABC. 토토사이트

    ReplyDelete
  32. Hi there! Nice post! Please tell us when I will see a follow up! Belgravia Ace Showflat

    ReplyDelete
  33. Thankyou for this wondrous post, I am glad I observed this website on yahoo. 토토사이트

    ReplyDelete
  34. Cool stuff you have got and you keep update all of us. 안전놀이터

    ReplyDelete
  35. You understand your projects stand out of the crowd. There is something unique about them. It seems to me all of them are brilliant. 온라인릴게임

    ReplyDelete
  36. Good artcile, but it would be better if in future you can share more about this subject. Keep posting. 안전놀이터

    ReplyDelete
  37. That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. check it

    ReplyDelete
  38. An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. 먹튀검증사이트

    ReplyDelete
  39. An fascinating discussion is value comment. I think that it is best to write extra on this matter, it won’t be a taboo topic however generally people are not enough to talk on such topics. To the next. Cheers 먹튀검증커뮤니티

    ReplyDelete
  40. You have outdone yourself this time. It is probably the best, most short step by step guide that I have ever seen. 먹튀검증

    ReplyDelete
  41. Someone Sometimes with visits your blog regularly and recommended it in my experience to read as well. The way of writing is excellent and also the content is top-notch. Thanks for that insight you provide the readers! best site to buy youtube views

    ReplyDelete
  42. Hi there! Nice stuff, do keep me posted when you post again something like this! research transcription services

    ReplyDelete
  43. Today, I was just browsing along and came upon your blog. Just wanted to say good blog and this article helped me a lot, due to which I have found exactly I was looking. 토토커뮤니티

    ReplyDelete
  44. I found your this post while searching for information about blog-related research ... It's a good post .. keep posting and updating information. 88카

    ReplyDelete
  45. Your website is really cool and this is a great inspiring article. 88카

    ReplyDelete
  46. CGood website! I truly love how it is easy on my eyes it is. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which may do the trick? Have a great day! 먹튀검증

    ReplyDelete
  47. RSJ is at the top of our game when it comes to tax-minimization strategies for asset growth, protection and transfer. We have always known that something was missing from traditional estate planning for high-net-worth families. click over here now

    ReplyDelete
  48. Mecanica utilizes the latest in cutting edge tools, technologies and methodologies in the areas of crash reconstruction and Event Data Recorders for documenting and analyzing accidents and related EDR data. pop over to this site

    ReplyDelete
  49. At Inovi, we’re committed to achieving the highest success rates while providing personalized care to our patients. Our Houston location offers expert care and is home to our state-of-the-art embryology lab. view publisher site

    ReplyDelete
  50. The Toto site legally operated in Korea is Batman Toto. Except for Batman Toto, all Toto sites are private Toto, and you can see that there are a lot of private Toto sites that do not have restrictions on the amount of bets and winnings, so it can be seen that there has been a rapid increase in recent years 먹튀검증

    ReplyDelete
  51. Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! daftar sbobet

    ReplyDelete
  52. Thank you so much as you have been willing to share information with us. We will forever admire all you have done here because you have made my work as easy as ABC. textbook solutions

    ReplyDelete
  53. You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. safe deposit box price

    ReplyDelete
  54. ดูหนังออนไลน์ ดูหนัง หนัง hd ซีรีย์เกาหลี ซีรีย์จีน ดูการ์ตูน ทีวีออนไลน์ ดูหนังออนไลน์ ฟรี nungsub หนังชนโรง Netflik ดูหนังออนไลน์ฟรี เว็บดูหนัง ดูฟรีไม่มีสะดุด ดูหนังออนไลน์

    ReplyDelete
  55. You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. iqama expiry

    ReplyDelete
  56. Thanks for picking out the time to discuss this, I feel great about it and love studying more on this topic. It is extremely helpful for me. Thanks for such a valuable help again. 팔팔카

    ReplyDelete
  57. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. renta car beograd

    ReplyDelete
  58. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. Dallas homicide defense lawyer

    ReplyDelete
  59. It is a great website.. The Design looks very good.. Keep working like that!. KissAnime Alternatives

    ReplyDelete
  60. Wow, this is really interesting reading. I am glad I found this and got to read it. Great job on this content. I like it. Dentitox Pro Reviews – After reading this, you should buy Dentitox Pro

    ReplyDelete
  61. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... nex777 login

    ReplyDelete
  62. i never know the use of adobe shadow until i saw this post. thank you for this! this is very helpful. 파워볼게임

    ReplyDelete
  63. Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. 토토사이트

    ReplyDelete
  64. I am very happy to discover your post as it will become on top in my collection of favorite blogs to visit. Film streaming

    ReplyDelete
  65. Liposuction should never be considered an alternative to a healthy lifestyle, it is one of the steps that can get you to a better version of yourself through fat reduction. Get More Info

    ReplyDelete
  66. I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. บาคาร่าออนไลน์

    ReplyDelete
  67. Excellent post. I was reviewing this blog continuously, and I am impressed! Extremely helpful information especially this page. Thank you and good luck. คาสิโนออนไลน์

    ReplyDelete
  68. I'm glad I found this web site, I couldn't find any knowledge on this matter prior to.Also operate a site and if you are ever interested in doing some visitor writing for me if possible feel free to let me know, im always look for people to check out my web site. nex777

    ReplyDelete
  69. It proved to be Very helpful to me and I am sure to all the commentators here! https://linklist.bio/togel

    ReplyDelete
  70. Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share. 대전마사지

    ReplyDelete
  71. I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it.. oxycontin

    ReplyDelete
  72. Thank you for this fascinating post, I am happy I observed this website on Google. Not just content, in fact, the whole site is fantastic. ฉีดฟิลเลอร์หน้าผาก อันตราย

    ReplyDelete
  73. I appreciate everything you have added to my knowledge base.Admiring the time and effort you put into your blog and detailed information you offer.Thanks. สิวอักเสบไม่มีหัว

    ReplyDelete
  74. Your company must be able to articulate your value proposition and address the challenges your prospects face on a daily basis in their business. If you can communicate how your products or services can help solve their pain points and help their business to grow, thrive, and sustain, your lead generation efforts will resonate with prospects in a meaningful way. best seller

    ReplyDelete
  75. Only strive to mention one's content can be as incredible. This clarity with your post is superb! Thanks a lot, hundreds of along with you should go on the pleasurable get the job done. textbook answers

    ReplyDelete
  76. It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. textbook answers

    ReplyDelete
  77. Good post but I was wondering if you could write a litte more on this subject? I’d be very thankful if you could elaborate a little bit further. Appreciate it! 먹튀검증

    ReplyDelete
  78. Your content is nothing short of bright in many forms. I think this is friendly and eye-opening material. I have gotten so many ideas from your blog. Thank you so much. https://yesmovies.agency

    ReplyDelete
  79. Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. https://myvumoo.com

    ReplyDelete
  80. Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. https://couchtuner.rest

    ReplyDelete
  81. Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. https://solar-movies.one

    ReplyDelete
  82. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here keep up the good work https://movieninja.live

    ReplyDelete
  83. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. https://the123movies.site

    ReplyDelete
  84. I am unable to read articles online very often, but I’m glad I did today. This is very well written and your points are well-expressed. Please, don’t ever stop writing. https://la-123movies.one

    ReplyDelete
  85. I have read your blog it is very helpful for me. I want to say thanks to you. I have bookmark your site for future updates. 먹튀검증

    ReplyDelete
  86. What a well written and compassionate article. I found your thoughts and wisdom to be encouraging and helpful. Cbd massage oil

    ReplyDelete
  87. I was looking at some of your posts on this website and I conceive this web site is really instructive! Keep putting up.. 대전건마

    ReplyDelete
  88. Thankyou for this wondrous post, I am glad I observed this website on yahoo. 대전건마

    ReplyDelete
  89. Thanks For Best Useful Information: STEP institute Interior Design Courses.https://www.iadm.edu.pk/academics/faculty-of-art-design/interior-design-course/

    ReplyDelete
  90. Thanks For Best Useful Information: STEP institute Interior Design Courses. https://www.iadm.edu.pk/academics/faculty-of-art-design/interior-design-course/

    ReplyDelete
  91. You have a good point here!I totally agree with what you have said!!Thanks for sharing your views...hope more people will read this article!!! 바둑이게임

    ReplyDelete
  92. For many people this is the best solution here see how to do it. 토토사이트

    ReplyDelete
  93. This is certainly as well a really good posting we seriously experienced looking through. It is far from on a daily basis we have risk to check out a little something. clothing sample makers uk

    ReplyDelete
  94. I found your this post while searching for some related information on blog search...Its a good post..keep posting and update the information. detective barcelona

    ReplyDelete
  95. You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. 토토사이트

    ReplyDelete
  96. I’ve been meditating on the identical issue personally recently. Pleased to see another person on the same wavelength! Nice article. 먹튀검증

    ReplyDelete
  97. You’ve got some interesting points in this article. I would have never considered any of these if I didn’t come across this. Thanks!. 꽁머니

    ReplyDelete
  98. Some truly wonderful work on behalf of the owner of this internet site , perfectly great articles . 꽁머니 커뮤니티

    ReplyDelete
  99. The way you write, you are really a professional blogger. latest news in urdu

    ReplyDelete
  100. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. 토토사이트

    ReplyDelete
  101. i was just browsing along and came upon your blog. just wanted to say good blog and this article really helped me. 토토사이트

    ReplyDelete
  102. Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return. 토토사이트

    ReplyDelete
  103. I really thank you for the valuable info on this great subject and look forward to more great posts. Thanks a lot for enjoying this beauty article with me. I am appreciating it very much! Looking forward to another great article. Good luck to the author! All the best! 먹튀검증

    ReplyDelete
  104. Thank you so much for such a well-written article. It’s full of insightful information. Your point of view is the best among many without fail.For certain, It is one of the best blogs in my opinion. Nagaland State Lottery

    ReplyDelete
  105. What a thrilling post, you have pointed out some excellent points, I as well believe this is a superb website. I have planned to visit it again and again. 토토사이트

    ReplyDelete
  106. Wow, What an Outstanding post. I found this too much informatics. It is what I was seeking for. I would like to recommend you that please keep sharing such type of info.If possible, Thanks. 먹튀검증

    ReplyDelete
  107. Good day! I just want to give a huge thumbs up for the great info you might have here on this post. I will be coming again to your weblog for extra soon. 스포츠중계

    ReplyDelete
  108. Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks slotxo

    ReplyDelete
  109. hi was just seeing if you minded a comment. i like your website and the thme you picked is super. I will be back. 오피사이트

    ReplyDelete
  110. I consider something really interesting about your site so I bookmarked . Giày jordan 1 low

    ReplyDelete
  111. If more people that write articles involved themselves with writing great content like you, more readers would be interested in their writings. I have learned too many things from your article. 오피사이트

    ReplyDelete
  112. This article was written by a real thinking writer. I agree many of the with the solid points made by the writer. I’ll be back. superslot

    ReplyDelete
  113. All your hard work is much appreciated. Nobody can stop to admire you. Lots of appreciation. แทงบอล

    ReplyDelete
  114. You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant! ไมเซล่า

    ReplyDelete
  115. This is certainly as well a really good posting we seriously experienced looking through. It is far from on a daily basis we have risk to check out a little something.protechrise

    ReplyDelete
  116. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. 먹튀폴리스

    ReplyDelete
  117. I am very much pleased with the contents you have mentioned. I wanted to thank you for this great article. 먹튀검증

    ReplyDelete
  118. When you move to a new place or buy your own Bedsheet online Pakistan.Bedsheet online Pakistan

    ReplyDelete
  119. his is the excellent blog page for anyone who wants to know about this theme. You recognize a lot its virtually difficult to argue with you (not that I really would want…HaHa). You absolutely put a fresh spin on a subject matter thats been published about for many years. Wonderful things, just excellent! 토토커뮤니티

    ReplyDelete
  120. birthday wishes for mother in law Birthday is the perfect occasion to show your love to mother in law. You may check our best collection of birthday wishes for her.

    ReplyDelete
  121. Thank you ever so for you blog article. Thanks Again. Really Great.
    share files

    ReplyDelete
  122. There is noticeably a bundle to know about this. I assume you made certain nice points in features also . Cab in Airdrie

    ReplyDelete
  123. Enjoying a luxurious lifestyle begins today. Life is too short to possibly be spent on things that are not beneficial. https://www.buyyoutubeviewsindia.in/youtube-marketing/

    ReplyDelete
  124. It very well may be a little one, several hundred dollars however it isn't zero. They likewise require prospecting and conversing with individuals to fabricate a group and procure extra pay. That organization showcasing implies. https://www.buyyoutubesubscribers.in/

    ReplyDelete
  125. This is very appealing, however , it is very important that will mouse click on the connection: Canada Visa for Quebec

    ReplyDelete
  126. It is a fantastic post – immense clear and easy to understand. I am also holding out for the sharks too that made me laugh. start a credit card processing business

    ReplyDelete
  127. Nice post mate, keep up the great work, just shared this with my friendz online file share

    ReplyDelete
  128. There is noticeably a bundle to know about this. I assume you made certain nice points in features also . Carstairs cab airport

    ReplyDelete
  129. Nice post. I was checking constantly this blog and I’m impressed! Extremely useful info specially the last part I care for such information a lot. I was seeking this certain info for a long time. Thank you and good luck.collapsed sewer line

    ReplyDelete
  130. Pretty nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed browsing your blog posts. After all I’ll be subscribing to your feed and I hope you write again soon! North American Bancard ISO

    ReplyDelete
  131. Termite Control service sargodha At Tahir termite control service, we never let the bed bugs to suck your blood and cause you trouble. It is essential for you to get rid of them and live a peaceful life. We are offering the ultimate bed-bug treatment in Lahore.Deemak Control sargodha

    ReplyDelete
  132. Kbc official website 2022 Presently KBC has situated different procedures to Check KBC Lottery Number with no cerebral pain. You may do your lottery wide assortment attestation via yourself. You can download the versatile form utility. Jio lottery office number 2022

    ReplyDelete
  133. Pretty nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed browsing your blog and come my web site info for you 토토사이트

    ReplyDelete
  134. Pretty nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed my web site info for you 토토사이트

    ReplyDelete
  135. An interesting dialogue is price comment. I feel that it is best to write more on this matter, it may not be a taboo topic however usually individuals are not enough to talk on such topics. To the next. Cheers. new internet device

    ReplyDelete
  136. When you choose same day delivery wales couriers services in uk with bubzycouriers, you can either select a one-off delivery or arrange for a more regular weekly or monthly courier service. No matter what you are after, request a free quote or a call back from us and we can get in touch to discuss your delivery requirements. wales couriers services in uk

    ReplyDelete
  137. Jio lottery office number 2022 This is the greatest upsetting question that has been mentioned via our fans pretty often. Kbc all india sim lucky drow There are such countless artificial sites on the web and it has wound up extreme for the competitors to comprehend the genuine Kbc lottery winner redirection. As you mindful, all Sims could be now enlisted inside the Kbc official website 2022. So the significant obstruction for the clients is to find their lottery range inside the rundown of champs.

    ReplyDelete
  138. Terrific review, among the list of much better articles Truly go through these days on the net, I recommend to all or any. Dentitox pro review

    ReplyDelete
  139. BubzyRecruitment is the construction staffing agency of choice for world-leading infrastructure businesses.construction recruitment agencies Salisbury

    ReplyDelete
  140. icd 10 anemiaThe Florida Gulf Coast University comes under one of the top-ranking universities in Southwest Florida. The university offers graduate and undergraduate programs for the students. To ease the learning process for the students, they introduced the fgcu canvas

    ReplyDelete
  141. BubzyRecruitment consultants have unparalleled recruiting expertise and an in-depth knowledge of the UK construction jobs market.construction recruitment agencies Dacorum

    ReplyDelete
  142. At BubzyPainting, we've qualified painters and decorators with over 30 years of experience to provide quality painting and decoration services.Painting Services in Arun

    ReplyDelete
  143. Ultimately, that decision is down to you. Based on the information above, you can look at your goods and deliveries.delivery near abberley

    ReplyDelete
  144. how tall is 50 cent Anemia is a health condition that lacks healthy and enough red blood cells. The red blood cells carry oxygen to the tissues of the human body. However, anemia is a dangerous disease. It makes the human body weak and tired.dan pena net worth

    ReplyDelete
  145. fgcu canvas If you aren’t a student at the FGCU, you may be wondering what this canvas is all about. To introduce the ease of learning for students and teachers, they have raised the learning management system. This is done to ensure everyone gets the flexibility of learning online.icd 10 anemia

    ReplyDelete
  146. I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. เกมสล็อต

    ReplyDelete
  147. Really a great addition. I have read this marvelous post. Thanks for sharing information about it. I really like that. Thanks so lot for your convene. buy cialis 40mg in

    ReplyDelete
  148. This is the type of information I’ve long been trying to find. Thank you for writing this information. detectives infidelidades

    ReplyDelete
  149. It is a fantastic post – immense clear and easy to understand. I am also holding out for the sharks too that made me laugh. 토토사이트

    ReplyDelete
  150. Positive site, where did u come up with the information on this posting 먹튀검증

    ReplyDelete
  151. NordVPN Premium Crack PC has become our first official cybersecurity partner. With such plenty of data shared continuously online! Nord VPN Cracked

    ReplyDelete
  152. Romantic Good Night Messages for Him or Her. Add spice to your romance. Let your special lady/man know you're thinking about her/him tonight.Is there anything.https://wishesquotz.com/good-night-wishes-for-sister/

    ReplyDelete
  153. CoffeeCup Web Form Builder Lite is, as the name hints at, a software utility which helps individuals create all sort of web forms. Coffeecup Form builder

    ReplyDelete